Okay
  Public Ticket #2449620
Large Number of Rows Returned
Closed

Comments

  •  2
    Andrew Kenth started the conversation

    I have over 250,000 records that need to be returned and filtered for use in a chart however the table seems to take a long time or just does not load. Are there any recommended alternatives of doing this to generating a query then filtering the table based on a parameter value? 

  •  2
    Andrew Kenth replied

    Digging further into the specific error I am seeing this in my logs:


    [23-Jun-2020 17:38:03 UTC] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /home2/objecto1/public_html/dataace/wp-content/plugins/wpdatatables/source/class.wpdatatable.php on line 2294


  •  2
    Andrew Kenth replied

    More Info. The query runs in 1 second using a remote connection leading me to believe that TMS cannot handle this amount of records. 

    * Affected rows: 0  Found rows: 252,956  Warnings: 0  Duration for 1 query: 0.094 sec. (+ 1.000 sec. network) */

  •  2,507
    Aleksandar replied

    Hello Andrew

    Sorry for the late response.

    There are wpDataTables that work with millions of entries, so I don't believe that's the case.

    You may want to increase the following:

    upload_max_file size, 

    post_max_size,

    max_execution_time

    You should be able to find these values in your php.ini file or on your hosting panel. If you can't find it yourself please ask your hosting provider to provide it.

    Other than that, you may want to look at your server's performance, hosting plan and the amount of other data you have on your pages. SQL query based tables (that don't pull ALL data onto a page) should work normally. Maybe a bit slower than non-server-side (smaller) tables, but work all-in-all.

    Kind Regards, 

    Aleksandar Vuković
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  •  2
    Andrew Kenth replied

    Thanks Aleksandar. Those settings did not change the behavior. Is there any way I can pass a url paremeter into the where statement of the sql that is executed? 

    I know I can pass a url parameter to filter a dataset (that's what we are doing now) but that requires pulling all data then filtering. 
    I also know we can use placeholders which helps reduce the number of tables required but shifts us to have multiple pages instead of multiple tables which is really no gain on our part. Is there any way to pass a url parameter into a placeholder? 

  •  2,507
    Aleksandar replied

    Hello Andrew

    Using the built-in features of the plugin will not be able to do this, however there is a workaround using this hook:

    wpdatatables_before_get_table_metadata( $tableId )

    Although, there is a bug on our end, and I'll have to ask you to bear with me to get it working properly.

    The action is called correctly when modifying a table in the backend, but if you have the same table rendered from the front-end the action is not called correctly.

    If you change a global variable within this action - works in the backend, in the front-end the default value is used:

    function testSetVar1toMNR($MyTable){
      global $wdtVar1;
      //get Column from Plugin
      $lMNR = do_shortcode('[ihc-user field="CUSTOM_FIELD_column"]');
      //Found?
      if ($lMNR>0) {
       $wdtVar1 = $lMNR; //Set VAR1 custom parameter to column
      }  else  {
          $wdtVar1 = "9999";
     //Default to 9999 --> needed for Admin Context
         }
           }
    add_action('wpdatatables_before_get_table_metadata', 'testSetVar1toMNR');
    

    Unfortunately at the moment I can offer only this Hot-Fix with non desired-side-effects:

    public function fillFromData($tableData, $columnData) {
            if (empty($tableData->table_type)) {
                return;
            }
            global $wdtVar1, $wdtVar2, $wdtVar3;
        //test: Change 31.07.2019 to update placeholders in production mode
      //needs default value for admin-context, otherwise breaks table creation
      do_action('wpdatatables_before_get_table_metadata');
            // Set placeholders
            $wdtVar1 = $wdtVar1 === '' ? $tableData->var1 : $wdtVar1;
    

    ….SNIP

    Here:  class.wpdatatable.php

    I hope this helps.

    After a new update is released, you won't need to add the hot-fix, though.

    Kind Regards, 

    Aleksandar Vuković
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  •  2,507
    Aleksandar replied

    Actually, Andrew,

    We've had a new update is released, so you won't need to add the hot-fix.

    The first part of the code goes into functions.php of your theme, while the other part of the code goes in ../wp-content/plugins/wpdatatables/source/class.wpdatatable.php. You can look for "public function fillFromData" in that file (should be around line 2722), and overwrite it.

    Please save a copy of the original file, just in case something goes wrong, so you can restore it.

    To be able to pass variables dynamically through URL, you should try something like this:

    6887824208.png

    And in the browser:

    2968087226.png

    That would return "10" and overwrite the original %VAR1% value.


    Kind Regards, 

    Aleksandar Vuković
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  •  2
    Andrew Kenth replied

    Sorry, but I'm not understanding what goes into functions.php and what I'm overwriting in public function fillFromData from the screenshot below. 

  •  2,507
    Aleksandar replied

    Hi again Andrew

    This goes into functions.php:

    function testSetVar1toMNR($MyTable){
      global $wdtVar1;
      //get Column from Plugin
      $lMNR = do_shortcode('[ihc-user field="CUSTOM_FIELD_column"]');
      //Found?
      if ($lMNR>0) {
       $wdtVar1 = $lMNR; //Set VAR1 custom parameter to column
      }  else  {
          $wdtVar1 = "9999";
     //Default to 9999 --> needed for Admin Context
         }
           }
    add_action('wpdatatables_before_get_table_metadata', 'testSetVar1toMNR');
    

    Since the update was released, the second code is no longer necessary.

    Kind Regards, 

    Aleksandar Vuković
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  •  2
    Andrew Kenth replied

    I have upgraded and entered the following but I'm getting the error that no data is returned in the table. It appears that quotes that are need to select strings are not being added to %VAR1% parameter. 


    Rendered query: SELECT geo, parent FROM WorldStats WHERE Parent = Wisconsin LIMIT 10
    MySQL said: Unknown column 'Wisconsin' in 'where clause'

  •  2,507
    Aleksandar replied

    Hi Andrew

    Please provide me a temporary WP-admin login for your site where this happens, so we could log in and take a look ‘from the inside’ as that’s the most efficient way to see and resolve the issue. We do not interfere with any data or anything else except for the plugin (in case that’s a production version of the site), and of course we do not provide login data to third party. 

    You can write credentials here just check Private Reply so nobody can see them except us.

    Kind Regards, 

    Aleksandar Vuković
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  •   Andrew Kenth replied privately
  •  2,507
    Aleksandar replied

    Hi Andrew.

    Thank you for the credentials. However, they are not Administrator level credentials:

    3461199706.png

    Can you please make that an admin role, and let me know?

    Kind Regards, 

    Aleksandar Vuković
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  •  2
    Andrew Kenth replied

    My apologies. Updated them. 

  •  471
    Isidora replied

    Hi Andrew.

    Aleksandar is out of office today and I will help you out.

    We check your website, and we notice that you only copy and paste code for hook that is the example from other customer needs which is the reason why your tables with placeholder VAR1 is not working. (attachment)

    Another this that we found in table with id 88 is that you are using in condition Parent instead of parent. (attachment)

    Please note that using hooks and filters requires some programming skills so you can adopt functionality for your needs.

    If you want to filter the table with URL parameter you need to change hook example that Aleksandar provide to you.

    Here are some advices in comments

    function setVar1fromURL($tableID){
        global $wdtVar1;
        //get URL parametar
        $varFromURL = $_GET['wdtVAR_1'];
        // Here is the condition that ask is that URL parameter set
        // Code like this will be executed for every table with SQL query
        // If you need for specific table then you will need to provide additional code
        // for example you want to filter only table with id 1 then
        // you will replace  if (isset($varFromURL)) { with this  if (isset($varFromURL && $tableID == 1)) {
        if (isset($varFromURL)) {
            $wdtVar1 = $varFromURL; //Set VAR1 with parameter from URL
        }
        // if your condition is not true then this code below will be executed
        // reason why your tables is not working because $wdtVar1 is always set to 9999
        // if URL parameter wdtVAR_1 is not set
        // if you don't need it you can remove it
        // This is example for some other customer needs so you need to adopt for your needs
        else {
            $wdtVar1 = "9999"; //Default to 9999 --> needed for Admin Context
        }
    }
    add_action('wpdatatables_before_get_table_metadata', 'setVar1fromURL');


    Kind Regards, 

    Isidora Markovic

    wpDataTables: FAQFacebookTwitterFront-end and back-end demoDocs

    Amelia: FAQFacebookTwitter |  Amelia demo sites | Docs

    You can try our wpDataTables add-ons before purchase on these sandbox sites:

    Powerful FiltersGravity Forms Integration for wpDataTablesFormidable Forms Integration for wpDataTables

  •   Andrew Kenth replied privately
  •  2,507
    Aleksandar replied

    Hi again Andrew

    Can you please share the code you've added with us?

    As a reminder - you cannot simply copy the code. You need to adjust it for your needs. The code we've provided is just an example.

    Kind Regards, 

    Aleksandar Vuković
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  •  2
    Andrew Kenth replied

    My need is very simple for now. I have tried both of the below examples in functions.php. It appears that there is no element named table in the $parsedQuery['FROM'][0] in class.wpdatatable.php.

    The functions error is likely happening in edit because there is no wdtVAR_1 here and it should instead load the default defined in Placeholders.  It's as if isset is not acting as expected. 


    class.wpdatatable.php on line 1501
      $tableName = $parsedQuery['FROM'][0]['table'];

    functions.php on line 167
    if (isset($varFromURL) && ($tableID == 93 || $tableID == 94 || $tableID == 99)) {
    if (isset($varFromURL)) {

    All additional Functions.php code

    function setVar1fromURL($tableID){
        global $wdtVar1;
        //get URL parametar
        $varFromURL = $_GET['wdtVAR_1'];
        // Here is the condition that ask is that URL parameter set
        // Code like this will be executed for every table with SQL query
        // If you need for specific table then you will need to provide additional code
        // for example you want to filter only table with id 1 then
        // you will replace  if (isset($varFromURL)) { with this  if (isset($varFromURL && $tableID == 1)) {   
    //if (isset($varFromURL) && ($tableID == 93 || $tableID == 94 || $tableID == 99)) {
    if (isset($varFromURL)) {
            $wdtVar1 = $varFromURL; //Set VAR1 with parameter from URL
        }
        // if your condition is not true then this code below will be executed
        // reason why your tables is not working because $wdtVar1 is always set to 9999
        // if URL parameter wdtVAR_1 is not set
        // if you don't need it you can remove it
        // This is example for some other customer needs so you need to adopt for your needs
        //else {
        //    $wdtVar1 = "9999"; //Default to 9999 --> needed for Admin Context
        //}
    }
    add_action('wpdatatables_before_get_table_metadata', 'setVar1fromURL');

  •  2,507
    Aleksandar replied

    Hi again Andrew

    It doesn't work in back-end because the GET parameter is not set.

    Remove this:

    $varFromURL = $_GET['wdtVAR_1'];

    And ask directly for GET:

    if (isset($_GET['wdtVAR_1'])) { $wdtVar1 = $_GET['wdtVAR_1']; }

    If the GET parameter isn't set per this code, VAR will be set to 9999, so you need to set the value you need, or remove it if you don't need the value for a GET parameter that isn't set.

    Kind Regards, 

    Aleksandar Vuković
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables

  •  2
    Andrew Kenth replied

    Thank you Aleksander, this is very helpful. It does resolve 1 of 2 errors thrown. However I am still receiving the error related to Undefined index: table which still crashes the edit page disallowing me from editing my page. 

    Notice: Undefined index: table in /home2/objecto1/public_html/dataace/wp-content/plugins/wpdatatables/source/class.wpdatatable.php on line 1501

    I'm wondering if I'm building my sql in such a way that it is throwing this error? 

    Example of most non normal sql on site:

    SSELECT
    NewCases_Geo,
    DailyCases_Geo,
    DailyCases.geoType,
    DailyCases,
    DailyCases_Change_p,
    DailyCases_Change_n,
    NewTests_Geo,
    DailyTests_Geo,
    DailyTests,
    DailyTests_Change_p,
    DailyTests_Change_n,
    geoTypeLookup
    FROM (
        SELECT 
            CONCAT(Geo," (",ValueChange,")") AS DailyCases_Geo,
            CONCAT(Geo," (",CurrentValue,")") AS NewCases_Geo,
            Geo,    
            geoType,
            CurrentValue AS DailyCases,        
            GREATEST(ValueChange, 0) AS DailyCases_Change_p,
            ABS(LEAST(ValueChange, 0)) AS DailyCases_Change_n,
            CASE SUBSTRING_INDEX(SUBSTRING_INDEX(uniqueID_Parent,'_',-2),'_',1)
                WHEN 'State' THEN 'County'
                WHEN 'Country' THEN 'State'
                WHEN 'SubRegion' THEN 'Country'
                WHEN 'Region' THEN 'SubRegion'
            END AS geoTypeLookup    
        FROM COVID_Deltas
        WHERE uniqueID_Parent = '%VAR1%'
            AND Statistic = 'DailyCases'
            AND PeriodDate = '1Day'
    ) AS DailyCases LEFT JOIN (
        SELECT 
            Geo,
            CONCAT(Geo," (",ValueChange,")") AS DailyTests_Geo,
            CONCAT(Geo," (",CurrentValue,")") AS NewTests_Geo,            
            CurrentValue AS DailyTests,        
            GREATEST(ValueChange, 0) AS DailyTests_Change_p,
            ABS(LEAST(ValueChange, 0)) AS DailyTests_Change_n 
        FROM COVID_Deltas
        WHERE uniqueID_Parent = '%VAR1%'
            AND Statistic = 'DailyTests'
            AND PeriodDate = '1Day'
    ) AS DailyTests ON DailyTests.Geo = DailyCases.Geo
    WHERE geoTypeLookup = geoType
    
    function setVar1fromURL($tableID){
        global $wdtVar1;  
     
        if (isset($_GET['wdtVAR_1'])) {
            $wdtVar1 = $_GET['wdtVAR_1']; 
        }   
        else {
            $wdtVar1 = "Wisconsin_State_UnitedStatesofAmerica"; 
        }
    }
    add_action('wpdatatables_before_get_table_metadata', 'setVar1fromURL');
            
    
  •  2,507
    Aleksandar replied

    Hello again Andrew

    You're correct, it's because of the query. The parser is not able to parse it since it's too complicated.

    You should try to create a MySQL view, which would hold the query. Unfortunately, you wouldn't be able to use the placeholders in the view, because it wouldn't work. You would need to create something like:

    SELECT * FROM view WHERE some_column= '%VAR1%'

    Or something like that.

    Kind Regards, 

    Aleksandar Vuković
    [email protected]

    Rate my support

    wpDataTables: FAQ | Facebook | Twitter | InstagramFront-end and back-end demo | Docs

    Amelia: FAQ | Facebook | Twitter | InstagramAmelia demo sites | Docs | Discord Community

    You can try wpDataTables add-ons before purchasing on these sandbox sites:

    Powerful Filters | Gravity Forms Integration for wpDataTables | Formidable Forms Integration for wpDataTables | Master-Detail Tables