Welcome Guest! To enable all features please Login. New Registrations are disabled.

Notification

Icon
Error

Login


Options
Go to last post Go to first unread
Offline MarkSiddall  
#1 Posted : 11 October 2019 04:21:16(UTC)
MarkSiddall

Rank: Newbie

Groups: Registered
Joined: 11/10/2019(UTC)
Posts: 3

I am having trouble importing data from a csv file with unequal numbers of columns per line, as well as having trouble importing time and date in the following format, is anyone able to help ?

The only ways I have been able to solve this is to open in notepad, and do search and replaces on "-", "+00:00" and to remove the "T" (before the time), and then replace all these with ","
The 2nd problem is interpreting the blank entries which vary in number per line.... again using notepad to do a search and replace is slow and clunky.

2019-09-29T19:17:44+00:00,,8,513024,17.6,30,14,6.85,84,TRUE,FALSE
2019-09-29T18:50:29+00:00,,,0,0,0,15,6.975,0,FALSE,FALSE
2019-09-29T18:23:17+00:00,,,0,0,0,16,6.975,0,FALSE,FALSE
2019-09-29T17:56:08+00:00,,,0,0,0,18,6.825,0,FALSE,FALSE
........
...
..

I also have the problems of importing another type of csv file :
;;;;"";"30100f05802807d40e549a01";;;;;;"";"0000";"Limit";"N/A";"IrVT";"-139.00";"12.19";"142B";"-1.0";"3";"N/A";"2.509";"-139.00";"9.06";"2253";"-1.0";"2";"N/A";"0.822";"-143.00";"6.07";"2254";"-1.0";"2";"N/A";"1.657";;;;;;;;"1414";"2019-09-30 08:17:44"

and as par of this i need to process the hex data :"30100f05802807d40e549a01", again any help in importing and processing hex data would be appreciated.

Surely there is a way to read in such csv files and parse the data thru various rules before processing and graphing ??

Wanna join the discussion?! Login to your SMath Studio Forum forum account. New Registrations are disabled.

Offline alyles  
#2 Posted : 11 October 2019 20:23:30(UTC)
alyles


Rank: Advanced Member

Groups: Registered
Joined: 23/03/2016(UTC)
Posts: 275
United States

Was thanked: 91 time(s) in 57 post(s)
Hi Mark,

I don't think you're going to be able to accomplish what you want directly within SMATH. The importData function doesn't handle text well unless it is surrounded by quotation marks. Additionally, I could not get it to work with empty columns either.

You're best bet right now is to import to excel and parse the CSV that way. They also have a built in hex converter. SMath doesn't have anything natively, but you could probably write a function that take a hex string and converts it to whichever base you'd like.

Feel free to join the SMath Studio Users Discord Channel: https://discord.gg/PayZpJW
Offline Jean Giraud  
#3 Posted : 11 October 2019 21:43:16(UTC)
Jean Giraud

Rank: Guest

Groups: Registered
Joined: 04/07/2015(UTC)
Posts: 6,868
Canada

Was thanked: 980 time(s) in 808 post(s)
Originally Posted by: MarkSiddall Go to Quoted Post
I am having trouble importing data from a csv file

Alternately, build each data matrix manually.
Global Warming in on of those I constructed ... > ½ day.
Data set from source is *.dat, just numbers.
Offline sinneD  
#4 Posted : 11 October 2019 23:20:45(UTC)
sinneD


Rank: Advanced Member

Groups: Registered
Joined: 12/03/2011(UTC)
Posts: 101
Location: Chicago

Was thanked: 16 time(s) in 7 post(s)
in my experience, its easier to manipulate in a spreadsheet program instead of SMath.

do all your cleaning up and reformatting there, even convert it to hex strings then import it into smath.

you can also cleverly make the matrix syntax in excel to copy and past into smath where the list is simply ordered of the row and column values and last two values are number of rows and columns. here is an example-

mat(r1c1,r1c2,r2c1,r2c2,r3c1,r3c2,r4c1,r4c2,4,2)

for 4 rows and 2 columns
Join the SMath Studio Users Discord Channel: https://discord.gg/PayZpJW
Offline mkraska  
#5 Posted : 12 October 2019 01:16:11(UTC)
mkraska


Rank: Advanced Member

Groups: Registered
Joined: 15/04/2012(UTC)
Posts: 1,980
Germany

Was thanked: 1120 time(s) in 718 post(s)
As was already mentioned, you can import to xlsx using excel with comma as column delimiter and then import to SMath Studio.

BTW, there is a glitch in mouse handling of the matrix resizer. It seems to compute the mouse drag required for resizing based on width of the first column.

importtest.sm (4kb) downloaded 106 time(s).
importtest.png
test.xlsx (12kb) downloaded 62 time(s).
Martin Kraska

Pre-configured portable distribution of SMath Studio: https://smath.com/wiki/SMath_with_Plugins.ashx
Offline MarkSiddall  
#6 Posted : 14 October 2019 01:23:13(UTC)
MarkSiddall

Rank: Newbie

Groups: Registered
Joined: 11/10/2019(UTC)
Posts: 3

Thanks for all the suggestions.... I think the xlsx option will be manageable for now - thanks.
Shame the importData function is not a bit more forgiving.
Offline sinneD  
#7 Posted : 15 October 2019 19:42:26(UTC)
sinneD


Rank: Advanced Member

Groups: Registered
Joined: 12/03/2011(UTC)
Posts: 101
Location: Chicago

Was thanked: 16 time(s) in 7 post(s)
Mark-
How do you mean 'more forgiving'?
What is it that you are experiencing or it not doing that you are expecting?
Some of us might be suffering from expert-blindpot.
Join the SMath Studio Users Discord Channel: https://discord.gg/PayZpJW
Offline MarkSiddall  
#8 Posted : 16 October 2019 03:40:12(UTC)
MarkSiddall

Rank: Newbie

Groups: Registered
Joined: 11/10/2019(UTC)
Posts: 3

I was meaning that the csv import is quite restrictive in the type and format of data it accepts.
The particular csv file has a different number of null entries per line, and the dataimport.csv function just ignores these rather than filling the matrix column with something like "" or 0
It also seems to dislike mixed types of data : some columns text, and some numeric
I could handle the date and time if inherently able to be imported as string.

For this data analysis job I was trying to avoid the extra step of importing the csv to excel, massaging the data in excel and then saving before bringin into SMath.
Offline uni  
#9 Posted : 16 October 2019 10:35:10(UTC)
uni


Rank: Advanced Member

Groups: Registered, Advanced Member
Joined: 10/11/2010(UTC)
Posts: 1,494
Man
Russian Federation

Was thanked: 1274 time(s) in 745 post(s)
You can use the Script component to parse any file as you want. But this will require some programming knowledge.
Russia ☭ forever
Viacheslav N. Mezentsev
Offline Jean Giraud  
#10 Posted : 18 October 2019 16:50:42(UTC)
Jean Giraud

Rank: Guest

Groups: Registered
Joined: 04/07/2015(UTC)
Posts: 6,868
Canada

Was thanked: 980 time(s) in 808 post(s)
Originally Posted by: MarkSiddall Go to Quoted Post
I was meaning that the csv import is quite restrictive in the type and format of data it accepts.
The particular csv file has a different number of null entries per line, and the dataimport.csv function just ignores these rather than filling the matrix column with something like "" or 0
It also seems to dislike mixed types of data : some columns text, and some numeric
I could handle the date and time if inherently able to be imported as string.

For this data analysis job I was trying to avoid the extra step of importing the csv to excel, massaging the data in excel and then saving before bringing into SMath.


Puzzling your comment ... maybe from punch card days.
The particular csv file has a different number of null entries per line, and the dataimport.csv function just ignores these rather than filling the matrix column with something like "" or 0

like "" or 0 ... or NaN

CSV.PNG
Offline drakon  
#11 Posted : 31 October 2019 15:35:10(UTC)
drakon

Rank: Newbie

Groups: Registered
Joined: 16/01/2018(UTC)
Posts: 1

Hello,

I have also troubles with importing a csv file. It works fine with numbers:

Code:

1	1
1	2


But doesnt work with strings:
Code:

"1"	"1"
1	2
Users browsing this topic
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You cannot vote in polls in this forum.