Hello Tim.

You can do it with an action

add_filter( 'wpdatatables_before_frontend_edit_row', 'Tim_filter', 10, 3 );
function Tim_filter( $formdata, $returnResult, $tableID) { 
    // Provide id for your table where you need validation, 
    // we insert 1 as example 
    if ($tableID == 1) { 
        // You will replace orig_header with your real name of orig_header 
        // which you can find it in column settings, and also a condition 
        // this is checker for specific column 
        if ($formdata['orig_header'] == ''){ 
            // You can provide your custom error message 
            $returnResult['error'] = __('Column name can not be empty', 'wpdatatables') ;
            echo json_encode($returnResult); 
            exit(); 
        } 
        // $formdata is array so you can also create a loop if you need 
        // to go through all form data entries and check if they pass your condition 
        //     foreach ($formdata as $signleEntry){ 
        //         if ($signleEntry == 'your condition'){ 
        //             $returnResult['error'] = __('Your custom error message', 'wpdatatables') ;
        //             echo json_encode($returnResult); 
        //             exit(); 
        //         } 
        //     } 
    }
}

This is only an example for validation.

If you want only to filter, then you can use wpdatatables_filter_frontend_formdata which can be seen here.