Okay
  Public Ticket #1475064
Creating large data table
Closed

Comments

  • mikesmith5656 started the conversation

    I am attempting to create a large data table with the WPdatatables plugin. The data is about 10,000,000 rows of data such as business name, address, phone number, city, state, etc. I have these files in 10 different CSV files, about 1,000,000 each. I can make one large file, or 10 different ones, either way.

    I have uploaded one of the CSV files to my media, but I'm noticing that the site times out when I attempt to upload this data into my plugin. Copy and pasting only allows about 50 rows per, which won't work.

    Is there an easy way for me to create this large data table? Connecting to a MySQL database appears to be my only option left, but I'm not very familiar with MySQL.

    What is the best way for me to proceed? Any direction, advice, and educational material you could point me to would be very much appreciated!

    Thank you!

  •  1,708
    Miloš replied

    HI mikesmith5656,
    Thank you for your purchase.

    When you are working with that large amount of data , the best option will be to use MySQL.
    Import that large file in your database and then make a query from it ( for example SELCT * FROM largeTable)

    If you want to use CSV file then maybe the other option is to open a table in Excel Like editor where you can paste more then 50 rows. ( attachment ) But this is something that is not recommended when you are working with 10 mil rows.

    You can also take a look at our documentation about very large tables so I hope this can help you out.


    Best regards.

    Kind Regards, 

    Miloš Jovanović
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  • mikesmith5656 replied

    Bodgan,

    Thank you! I was able to follow the instructions for a very large table via your youtube video "How to work with very large tables in Wordpress with wpDataTables". I converted my CSV to SQL, imported it into phpMyAdmin, and then created a data table by connecting it to the MySql.

    When I "create new table" --> "Create a table linked to an existing data source" --> select my SQL Query, the page waits a few seconds and then gives me a red error popup that says internal server error. Despite the error message, it does create the correct data table, which shows up in my "wpDataTables". If I click to view the data table from the WP dashboard, the page just shows up blank.

    When I preview the data table, it does work (Please see sreenshots for examples). But I have no access to modify the table settings. For instance, I need to enable horizontal scrolling. Do you have any any ideas how I can fix the issue of not being able to see the data table from the WP dashboard?

    Thank you! 

  •  1,708
    Miloš replied

    HI mikesmith5656,

    The 500 Internal Server Error is a server-side error, meaning the problem probably isn't with your computer, Internet connection or plug ins but instead is a problem with the web site's server.

    Can you please investigate your server log and see if there is some errors.


    Best regards.

    Kind Regards, 

    Miloš Jovanović
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  • mikesmith5656 replied

    Hey Bogdan,

    I went to cpanel --> error logs, and took a screenshot for you (Attached). There doesn't appear to be any errors logged.


  •  1,708
    Miloš replied

    HI mikesmith5656,

    Please provide me a temporary WP-admin login for your site where this happens, so we could log in and take a look ‘from the inside’ as that’s the most efficient way to see and resolve the issue. We do not interfere with any data or anything else except for the plugin (in case that’s a production version of the site), and of course we do not provide login data to third party. You can write credentials here just check Private Reply so nobody can see them except us.

    Also point me to the table ID so I can take a look.


    Best regards.

    Kind Regards, 

    Miloš Jovanović
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  •   mikesmith5656 replied privately
  •  1,708
    Miloš replied

    Hi mikesmith5656,

    This is the error that I get when I debug further ( attachment) 

    This is probably happening because of the possible values in the table that are set for filtering, there are too many of them.
    Ok, next steps are that if you can send me that file from which you created a table, sql or zipped so I can import it on my database and investigate on our side.

    Also can you please tell me the values that you set for memory_limit, post_max_size, max_execution_time  and max_input_time?

    This is our set

    memory_limit = 256M 

    post_max_size = 128M 

    max_execution_time = 300

    max_input_time = 300


    Best regards.

    Kind Regards, 

    Miloš Jovanović
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  •   mikesmith5656 replied privately
  •  1,708
    Miloš replied

    Hi mikesmith5656

    After some testing we managed to create a table, but with some limitation.(attachment) We are working on improvement for making tables with large number of data, so for now you have to comment some parts of code to make this possible. You will have limitation only for Predefined value for filtering and editing. 

    You can using filtering (for filtering filter type:text, number, number range, date range, datetime range and time range) and editing features( Editor type:you can use all except Single value selectbox and Multi-value selectbox)  of table but you can't use predifined values for checkbox, selectbox and multiselectbox filter or editor. 

    Predifined values are taking distinct values from column so if you have 100 it's not a problem, but you have at least 100000 or even more.

    You can read about this in our documentation on this link(look for Predefined default filter values, Checkbox filter, Multiselectbox and Select box filter)

    We made some  changes in code in file 

    wp-content/plugins/wpdatatables/source/class.wdtconfigcontroller.php around line 60

    you can find this:

    foreach ($tableData->columns as &$column) {
       $column->defaultValueValues = $wpDataTable->getColumn($column->orig_header)->getDefaultValues();
    }

    and you have to commented that like this:

    /*foreach ($tableData->columns as &$column) {    $column->defaultValueValues = $wpDataTable->getColumn($column->orig_header)->getDefaultValues();
    }*/

    Also this line in file : 

    /wp-content/plugins/wpdatatables/assets/js/wpdatatables/wpdatatables.js  around line 1030

    you can find this:

    for (var i in tableDescription.advancedEditingOptions.aoColumns) {    var defaultValue = tableDescription.advancedEditingOptions.aoColumns[i].defaultValue;    var editorInputType = tableDescription.advancedEditingOptions.aoColumns[i].editorInputType;    if (defaultValue) {        if ($.inArray(editorInputType, ['selectbox', 'multi-selectbox']) !== -1) {            defaultValue = editorInputType == 'multi-selectbox' ? defaultValue.split('|') : defaultValue;            $('#wdt-frontend-modal .editDialogInput:not(.bootstrap-select):eq(' + i + ')').selectpicker('val', defaultValue);        } else {            $('#wdt-frontend-modal .editDialogInput:not(.bootstrap-select):eq(' + i + ')').val(defaultValue);        }    }
    }

    and you have to commented that like this:

    /*for (var i in tableDescription.advancedEditingOptions.aoColumns) {    var defaultValue = tableDescription.advancedEditingOptions.aoColumns[i].defaultValue;    var editorInputType = tableDescription.advancedEditingOptions.aoColumns[i].editorInputType;    if (defaultValue) {        if ($.inArray(editorInputType, ['selectbox', 'multi-selectbox']) !== -1) {            defaultValue = editorInputType == 'multi-selectbox' ? defaultValue.split('|') : defaultValue;            $('#wdt-frontend-modal .editDialogInput:not(.bootstrap-select):eq(' + i + ')').selectpicker('val', defaultValue);        } else {            $('#wdt-frontend-modal .editDialogInput:not(.bootstrap-select):eq(' + i + ')').val(defaultValue);        }    }
    }*/

    Also this line of code was commented in file

    wp-content/plugins/wpdatatables/source/class.wdtconfigcontroller.php around line 102

    /*foreach ($tableData->columns as &$column) {
                    $column->defaultValueValues = $wpDataTable->getColumn($column->orig_header)->getDefaultValues();
    if ($column->defaultValueValues) {
                        sort($column->defaultValueValues);
                    }
                }*/



    Please let us now  is everything working fine when you test it out.


    Best regards.

    Kind Regards, 

    Miloš Jovanović
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  • mikesmith5656 replied

    Bogdan, this appears to work, thank you!

    https://usadatabase.com/

    Looks great!

  • mikesmith5656 replied

    Bogdan,

    Whenever I attempt to import a new zipped .sql database into the phpMyAdmin, it will only import about 100 rows. I have tried re-generating a few .sql files, zipping them up and importing them, every time it will only allow ~100 rows. Did any of the settings change that would not allow the import the entire file?

    Thanks for your help!

    -- Mike

  •  1,708
    Miloš replied

    HI Mike,

    Can you please tell me is it happening in phpmyadmin or in Wordpress.
    Change of the code that is applied is not related to phpmyadmin.


    Best regards.

    Kind Regards, 

    Miloš Jovanović
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables