Filament•4mo ago

Filament's Export function creating duplicates.

TL;DR : Filament's export action creating duplicates when exporting larger record sets. I have a very simple model, resource, and exporter. However when I try to export any more than a hundred or so, it mangles the data and returns duplicate records. I am using : laravel/framework: ^10.10 filament/filament: ^3.1 (3.3.20) Originally I was using pxlrbt/filament-excel and that was doing the same thing, so I converted over to using the inbuilt. Which is where I realised the inbuilt one is also doing the same thing. Model:

namespace App\Models;
use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
class Activity extends Model
{
    use HasFactory;
    protected $guarded = [];
}

namespace App\Models;
use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;
class Activity extends Model
{
    use HasFactory;
    protected $guarded = [];
}

Migration:

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
    public function up(): void
    {
        Schema::create('activities', function (Blueprint $table) {
            $table->id();
            $table->string('activities_title')->nullable();
            $table->date('start_date');
            $table->timestamps();
        });
    }
    public function down(): void
    {
        Schema::dropIfExists('activities');
    }

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;
return new class extends Migration
{
    public function up(): void
    {
        Schema::create('activities', function (Blueprint $table) {
            $table->id();
            $table->string('activities_title')->nullable();
            $table->date('start_date');
            $table->timestamps();
        });
    }
    public function down(): void
    {
        Schema::dropIfExists('activities');
    }

I'll add the resource and exporter as comments.

Solution:

Updating here as well. If I am using Dennis' plugin I can just do this and it fixes the duplication. ```swift ExportBulkAction::make() ->exports([...

Jump to solution

25 Replies

403gtfoOP•4mo ago

Resource:

namespace App\Filament\Resources;

use App\Filament\Exports\ActivityExporter;
use App\Models\Activity;
use App\Filament\Resources\ActivityResource\Pages\ListActivities;
use Filament\Resources\Resource;
use Filament\Tables\Columns\TextColumn;
use Filament\Tables\Table;
use Filament\Tables\Actions\ExportAction;

class ActivityResource extends Resource
{
    protected static ?string $model = Activity::class;
    protected static ?string $navigationIcon = 'heroicon-o-trophy';
    protected static ?string $navigationGroup = 'Activities';

    public static function tableFields()
    {
        return [
            TextColumn::make('id')->hidden(),
            TextColumn::make('start_date')
                ->searchable()
                ->sortable(),
            TextColumn::make('activities_title')
                ->searchable()
                ->sortable(),
        ];
    }
    public static function table(Table $table): Table
    {
        return $table
            ->columns(self::tableFields())
            ->defaultSort('start_date', 'desc')
            ->headerActions([
                ExportAction::make()
                    ->exporter(ActivityExporter::class)
            ]);
    }

    public static function getPages(): array
    {
        return [
            'index' => ListActivities::route('/'),
        ];
    }
}

namespace App\Filament\Resources;

use App\Filament\Exports\ActivityExporter;
use App\Models\Activity;
use App\Filament\Resources\ActivityResource\Pages\ListActivities;
use Filament\Resources\Resource;
use Filament\Tables\Columns\TextColumn;
use Filament\Tables\Table;
use Filament\Tables\Actions\ExportAction;

class ActivityResource extends Resource
{
    protected static ?string $model = Activity::class;
    protected static ?string $navigationIcon = 'heroicon-o-trophy';
    protected static ?string $navigationGroup = 'Activities';

    public static function tableFields()
    {
        return [
            TextColumn::make('id')->hidden(),
            TextColumn::make('start_date')
                ->searchable()
                ->sortable(),
            TextColumn::make('activities_title')
                ->searchable()
                ->sortable(),
        ];
    }
    public static function table(Table $table): Table
    {
        return $table
            ->columns(self::tableFields())
            ->defaultSort('start_date', 'desc')
            ->headerActions([
                ExportAction::make()
                    ->exporter(ActivityExporter::class)
            ]);
    }

    public static function getPages(): array
    {
        return [
            'index' => ListActivities::route('/'),
        ];
    }
}

Exporter:

class ActivityExporter extends Exporter
{
    protected static ?string $model = Activity::class;

    public static function getColumns(): array
    {
        return [
            ExportColumn::make('id'),
            ExportColumn::make('activities_title'),
            ExportColumn::make('end_date'),
        ];
    }

    public static function getCompletedNotificationBody(Export $export): string
    {
        $body = 'Your activity export has completed and ' . number_format($export->successful_rows) . ' ' . str('row')->plural($export->successful_rows) . ' exported.';

        if ($failedRowsCount = $export->getFailedRowsCount()) {
            $body .= ' ' . number_format($failedRowsCount) . ' ' . str('row')->plural($failedRowsCount) . ' failed to export.';
        }

        return $body;
    }
}

class ActivityExporter extends Exporter
{
    protected static ?string $model = Activity::class;

    public static function getColumns(): array
    {
        return [
            ExportColumn::make('id'),
            ExportColumn::make('activities_title'),
            ExportColumn::make('end_date'),
        ];
    }

    public static function getCompletedNotificationBody(Export $export): string
    {
        $body = 'Your activity export has completed and ' . number_format($export->successful_rows) . ' ' . str('row')->plural($export->successful_rows) . ' exported.';

        if ($failedRowsCount = $export->getFailedRowsCount()) {
            $body .= ' ' . number_format($failedRowsCount) . ' ' . str('row')->plural($failedRowsCount) . ' failed to export.';
        }

        return $body;
    }
}

Note I have tried: - blowing away the DB and using dummy data and it still generated duplicates. - adding protected static ?string $primaryKey = 'id'; still generated duplicates - Using both core and pxlrbt/filament-excel exporters: still generated duplicates - reverting the code back to about a year ago. still generated duplicates. - exporting only a few records (like 50) no duplicates.

Dennis Koch•4mo ago

I guess it must have something to do with pagination. Probably duplicate start date? Can you try adding id as a second sorting param?

toeknee•4mo ago

I wonder if it's down to Daylight savings

403gtfoOP•4mo ago

But you be right about the pagination and duplicate values making it sad. I: 1. converted the date() field to a timestamp 2. regenerated 1000 records 3. tested some exports with various sorting So far... no duplicates. But when I sort on another field that has to have duplicate values.... it exports duplicate records. Sorry lots of edits... so a partial solution for the date but still a bug when sorting by other fields it generates duplicates in the exports. I couldnt work out how to "Can you try adding id as a second sorting param?" how do I do that sorry?

403gtfoOP•4mo ago

I have created a demo project to test the bug if you like. https://github.com/uqjohart/duplication-bug-filament (While it has sail in it, you don't need sail, and just just deploy as normal) I included a db dump in the db-dump folder (all dummy data) filament use is test@test.com password To replicate: 1. sot by date that is now random timestamps -> export -> no duplicates 2. sort by name that WILL contain duplicates -> export -> duplicates Optional: Switch out the default filament exporter to @Dennis Koch 's export and the same behavior persists. I might have to submit this as a proper filament bug as well if we don't have an thoughts. My clients are jumping on my back about it :S

GitHub

GitHub - uqjohart/duplication-bug-filament

Contribute to uqjohart/duplication-bug-filament development by creating an account on GitHub.

403gtfoOP•4mo ago

I should of named that better >_>

403gtfoOP•4mo ago

https://github.com/filamentphp/filament/issues/16497

GitHub

Exporter exporting incorrectly and causing duplicates with non uniq...

Package filament/filament Package Version v3.3.21 Laravel Version ^12.0 Livewire Version ^3.6 PHP Version 8.3.21 Problem description When exporting a large volume of record (>100/200) the export...

Barry•4mo ago

Can you check this? https://github.com/filamentphp/filament/pull/16552

GitHub

Add orderBy qualified keyname for stable pagination sort by barryvd...

Makes sure to add a unique key to the orderBy to prevent missing/duplicate items when pagination because of unstable sorting, when keys are the same. Description This adds a default sort to pagina...

Barry•4mo ago

Not sure if that's the best solution though

403gtfoOP•4mo ago

I'll take a look first thing when I get to work in the morning. Appreciate y'all looking into the issue.

Barry•4mo ago

for me it was with pagination though, not sure if it uses the same pagination

403gtfoOP•4mo ago

I made the change to CanPaginateRecords but it doesnt seem to fix it in my test one :c Hmm... that being said you may have put me on to a fix (And I think it was what @Dennis Koch was telling me to do). I changed the factory to:

return [
  'name' => $this->faker->randomElement(['Short Activity Name', 'Another Short Name',]),
  'start_date' => $this->faker->dateTimeBetween('-2 year', '-1 year'),
];

return [
  'name' => $this->faker->randomElement(['Short Activity Name', 'Another Short Name',]),
  'start_date' => $this->faker->dateTimeBetween('-2 year', '-1 year'),
];

And the resource table column to:

 Tables\Columns\TextColumn::make('name')
     ->sortable(
         true,
         fn($query, $direction) => $query->orderBy('name', $direction)->orderBy('id', $direction)
     ),

 Tables\Columns\TextColumn::make('name')
     ->sortable(
         true,
         fn($query, $direction) => $query->orderBy('name', $direction)->orderBy('id', $direction)
     ),

And that seems to have killed the duplicates. Including using pxlrbt\FilamentExcel It is a PITA to go through all my resources to update, but if it work I am happy until it gets resolved in core. I need to test on my larger project. Works on direct columns... but not relational columns such as activityType.name Especially when activityType.name can be null. Then it is al sorts of messed up.

Dennis Koch•4mo ago

I don’t think this one is a core issue. If you have an order on a date and you have multiple records with the same date, there is no guarantee that they are sorted the same every time.

403gtfoOP•4mo ago

Problem is it happens on every field (not just date) that can be non unique :S My fix doesnt work on relational data such as the activityType.name... if I can get that working I'll be golden. Seeing as it is breaking on that little clean install app, I cant see it be anything other than being in core as that one has nothing from my larger app other than the simple activity resource and bits. This bug is seriously making me regret my life choices.

Dennis Koch•4mo ago

It's not a Core issue. It's a database limitation.

Barry•4mo ago

Maybe the export is not a core issue but for the Table component, Filament should handle that Imho

Dennis Koch•4mo ago

What's your recommendation? Always put a sort by id on the table?

Dennis Koch•4mo ago

https://github.com/filamentphp/filament/issues/16497 😅

GitHub

Exporter exporting incorrectly and causing duplicates with non uniq...

Package filament/filament Package Version v3.3.21 Laravel Version ^12.0 Livewire Version ^3.6 PHP Version 8.3.21 Problem description When exporting a large volume of record (>100/200) the export...

Barry•4mo ago

I think Dan said he did something similar In v4

Barry•4mo ago

But for v3 something like this perhaps https://github.com/filamentphp/filament/pull/16556

GitHub

Make stablesort configurable by barryvdh · Pull Request #16556 · ...

Makes sure to add a unique key to the orderBy to prevent missing/duplicate items when pagination because of unstable sorting, when keys are the same. Description Re-does #16552 This adds a stableS...

Barry•4mo ago

If you need it to be optional

Dennis Koch•4mo ago

I see you already dived deeper into that.

Solution

403gtfo•4mo ago

Updating here as well. If I am using Dennis' plugin I can just do this and it fixes the duplication.

ExportBulkAction::make()
    ->exports([
        ExcelExport::make()
            ->fromTable()
            ->modifyQueryUsing(
                fn($query) => $query->orderBy('activities.id', 'asc')
            )
        ]),

ExportBulkAction::make()
    ->exports([
        ExcelExport::make()
            ->fromTable()
            ->modifyQueryUsing(
                fn($query) => $query->orderBy('activities.id', 'asc')
            )
        ]),

Seems so stupid obvious now in hindsight. I will take my award for being a spoon now please.

Dennis Koch•4mo ago

Here you go 🥄

403gtfoOP•4mo ago

Thank you thank you. I'd like to thank my parents, without whom I would never be able to have reached this level of spoonery.

Gaming

Programming

Filament's Export function creating duplicates.

Did you find this page helpful?