One of the queries that Synology makes

One of the queries that Synology makes results in a NextMarker loop - forever

31 Replies

Sid•3y ago

Is it making the same request over and over? Like, with the same delimiter/prefix/marker?

kianOP•3y ago

Yep R2 returns a Marker of foo and a NextMarker of foo so it keeps following NextMarker I'm going to move the files from my R2 bucket onto S3 & run the same request against S3 Specifically it's <NextMarker>Resilience_1.hbk/</NextMarker> (varies between upload jobs but you get the idea) Previously the job uploaded to a folder called sdk-example and it looped on this

    <Marker>sdk-example/</Marker>
    <MaxKeys>1000</MaxKeys>
    <NextMarker>sdk-example/</NextMarker>

    <Marker>sdk-example/</Marker>
    <MaxKeys>1000</MaxKeys>
    <NextMarker>sdk-example/</NextMarker>

It's basically being told 'to check this folder, use this marker' but that marker isn't progressing anything

kianOP•3y ago

kianOP•3y ago

so NextMarker is Resilience_1.hbk/ so I'll add the marker search param with that value

kianOP•3y ago

kianOP•3y ago

now I'm stuck forever, since it's just telling me to check where I already am

kianOP•3y ago

if I remove the delimiter and the marker, we can see there are files

kianOP•3y ago

including other folders after that

kianOP•3y ago

if I do the same against S3:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Name>sdk-example-kian</Name>
    <Prefix></Prefix>
    <Marker>Resilience_1.hbk/</Marker>
    <MaxKeys>1000</MaxKeys>
    <Delimiter>/</Delimiter>
    <EncodingType>url</EncodingType>
    <IsTruncated>false</IsTruncated>
</ListBucketResult>

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Name>sdk-example-kian</Name>
    <Prefix></Prefix>
    <Marker>Resilience_1.hbk/</Marker>
    <MaxKeys>1000</MaxKeys>
    <Delimiter>/</Delimiter>
    <EncodingType>url</EncodingType>
    <IsTruncated>false</IsTruncated>
</ListBucketResult>

Let me try and setup a repro that doesn't involve Synology Repro:

/* s3 client setup */

await S3.send(new PutObjectCommand({ Bucket: "sdk-example", Key: "folder/foldertwo/file" }));

console.log(
  await S3.send(
    new ListObjectsCommand({
      Bucket: "sdk-example",
      Delimiter: "/",
      Prefix: "",
      Marker: "folder/",
    })
  )
);

/* s3 client setup */

await S3.send(new PutObjectCommand({ Bucket: "sdk-example", Key: "folder/foldertwo/file" }));

console.log(
  await S3.send(
    new ListObjectsCommand({
      Bucket: "sdk-example",
      Delimiter: "/",
      Prefix: "",
      Marker: "folder/",
    })
  )
);

S3:

  Delimiter: '/',
  IsTruncated: false,
  Marker: 'folder/',
  MaxKeys: 1000,
  Name: 'sdk-example-kian',
  Prefix: ''

  Delimiter: '/',
  IsTruncated: false,
  Marker: 'folder/',
  MaxKeys: 1000,
  Name: 'sdk-example-kian',
  Prefix: ''

R2:

  CommonPrefixes: [ { Prefix: 'folder/' } ],
  Delimiter: '/',
  IsTruncated: false,
  Marker: 'folder/',
  MaxKeys: 1000,
  Name: 'sdk-example',
  NextMarker: 'folder/',
  Prefix: ''

  CommonPrefixes: [ { Prefix: 'folder/' } ],
  Delimiter: '/',
  IsTruncated: false,
  Marker: 'folder/',
  MaxKeys: 1000,
  Name: 'sdk-example',
  NextMarker: 'folder/',
  Prefix: ''

Both ran on empty buckets

kianOP•3y ago

kianOP•3y ago

Same file/folder path on both To clarify why Marker: "folder/" is used, that's what R2 returns as NextMarker if I omit the marker.

Sid•3y ago

I somewhat vaguely remember seeing an issue like this on our issue tracker But it might have been something completely different Okay but this is suuuper helpful, I can now write a test to see what happens

kianOP•3y ago

kianOP•3y ago

Let's just say Synology will, uh... ferrissweatmegaspin

Sid•3y ago

Good thing you have R2_NOCOST right

kianOP•3y ago

Yes - but we're also within free blobcatbusiness

itsmatteomanf•3y ago

@kiannh is great, yeah If you want I can keep going 😛 This varies as it’s the base folder name, which defaults to the NAS name with the hbk extension. You can select it.

kianOP•3y ago

I should apply as QA /s

James•3y ago

Glad this was got to the bottom of! Thanks so much for the debug y’all. Would love to use R2 with my synology at some point 👀

Sid•3y ago

Yeah I haven't dug into this yet, but if it is something simple on our end, expect a fix soon. Otherwise I might procrastinate, but I'll report back here what I find

itsmatteomanf•3y ago

I have my job ready, if you need a test just ping me anytime. It’s a very easy click to test, and even if I need to set-it up again I did a separate token, so I roll and restart easily. It was fun, seeing @kiannh swear at Signatures in S3 CLI 😂

kianOP•3y ago

James•3y ago

Any news here? 👀 Is there a better place I could file this for tracking? Ticket to custesc maybe?

Sid•3y ago

Yeah a CUSTESC is probably going to the best course of action, I haven’t found time to look into this yet 😦

James•3y ago

Alrighty, will do 🙂

itsmatteomanf•3y ago

Should we do one and CC the other into it? 👀

James•3y ago

can do. I was gonna take a look this weekend, don't have much time this week. Will DM you beforehand in case you've already done one lul

itsmatteomanf•3y ago

Not sure I’m gonna find the time, as it requires a bunch of info to share and to prepare 😅 @sdnts so, we should CUSTESC it, right? No progress in the meantime?

Sid•3y ago

Hey, yeah that sounds right. I've been OOO for the last couple of weeks, so no progress yet unfortunately

itsmatteomanf•3y ago

Hopefully gonna do that today, finally...

Gaming

Programming

One of the queries that Synology makes

Did you find this page helpful?