Okay
  Public Ticket #3037566
Maximum import size CSV table
Closed

Comments

  •  1
    Jouke started the conversation

    Best wpDataTables, 

    I have been testing with the sandbox, an tried to upload a CSV file with 100.000 records that includes 5 cells with value. (total of 500.000 cells) the total file size is 3,35mb. 

     I cant find any limit regarding uploaded data size except for the maximum upload size of 128mb.

    But when uploading the file, I’m getting this error :
    Unexpected response from the server. The file may have been uploaded successfully. Check in the Media Library or reload the page.

    The file will not upload.
    When I reduce the number of records to 240 than it will upload successfully.

    Can, and how can I upload larger amounts of data like this 100.000 records ?

  •  1
    Jouke replied

    Someone here to help ?

  •  1,846
    Miloš replied

    Hi, Jouke

    Firstly, I would like to sincerely apologize for the delayed response as we have been experiencing an unusually high number of tickets. I am sorry that it has taken longer than usual to respond to your concern and your patience is highly appreciated.

    - Just as you described, it is correct, because you cut down the number of rows to 240, we are able to import the CSV without issues.

    - Could you send us the full version of the CSV with all of the rows, and we will replicate the issues on our end?

    If it contains any kind of sensitive data, you can check the reply as Private, so nobody can see it but us.

    - Our plugin is capable of handling very large amounts of data using SQL server-side processing. 

    If you'd like, you could check out this example on  our documentation with a table where  wwe used a data set of 2.560.000 rows.

    On our sandbox, it might be that we did not set the upload limit high enough, we will certainly check that, too.

    Kind Regards, 

    Miloš Jovanović
    [email protected]

    Rate my support

    Try our FREE mapping plugin! MapSVG - easy Google maps, interactive SVG maps and floor plans, choropleth maps and much more - https://wordpress.org/plugins/mapsvg-lite-interactive-vector-maps/

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  •  1
    Jouke replied

    Thanks Milos,

    Well it seems to work now.
    I tryed to upload it last time to this post but it didn't work, maybe it was because i tested it from my holiday internet connection, but now i'm working from our office it seems to work fine. 

    i already managed to create a chart with it, so thanks for your support in this case. 

    I also have another question regarding the CSV upload. 
    Is it possible to create a table out of multiple CSV files ?

    We have a machine that will produce every day an CSV file to be uploaded, to a server. 
    I would like to create 1 table and chart out of all the CSV files.

  •  1,846
    Miloš replied

    Hi, Jouke

    Sorry for the delay.

    I am glad to see that you have now successfully started with the table, and a chart, as well.smile.png

    -

    If the original CSV file would be uploaded to a certain file path;

    and if the new CSV files would only overwrite this file, to keep the same path, and same column structure, such as header names;  and not to add or remove columns;

    basically only to edit existing rows, or update new rows with values, then it can work,

    if you use the method to link the CSV to wpDataTable output;

    -

    But if they are going to have 10 thousand rows, or more, like this one... basically anything more than 3000 rows will be extremely slow, or even break the page, unfortunately,

    in that case you would have to switch to something that uses server-side processing, such as SQL based tables.

    -

    Alternatively, if you import the CSV to create a table in the database;

    there are some workarounds, but they require a bit more work, such as :

    You could create an initial imported table from CSV;

    it will exist in the database;  but it won't be linked to the source CSV anymore;

    If you want to update it later using CSV,

    the only way is to either use something like PhPMyAdmin, to insert CSV as update;

    but you would have to remove headers, and remove old rows > only to keep the new/updated rows, and import CSV like that...

    -

    There are also custom SQL Queries that might achieve that, to update a table via CSV automatically from certain path ; 

    but both of these " custom workaround" solutions would not be covered by our support.

    -

    For SQL Queries, please note :

    Using this plugin feature requires at least a basic knowledge of SQL. It is assumed that you can create the table in some MySQL data manager (e.g., PHPMyAdmin, MySQL Workbench), and prepare a SQL query that will return the data you need. If you do not know how to use MySQL, refer to the “Create a MySQL-query-based table by generating a query to MySQL database” section. It will explain how to create complete tables or MySQL queries without this basic knowledge. Please be advised that preparing MySQL queries for you is not included in the plugin support.

    -

    Sorry for such a long message, i just wish to try and advise you in the best possible way with all of the details.

    Let me know how all that sounds. Thanks.

    Kind Regards, 

    Miloš Jovanović
    [email protected]

    Rate my support

    Try our FREE mapping plugin! MapSVG - easy Google maps, interactive SVG maps and floor plans, choropleth maps and much more - https://wordpress.org/plugins/mapsvg-lite-interactive-vector-maps/

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables