One of the queries that Synology makes

One of the queries that Synology makes results in a NextMarker loop - forever

31 Replies

Unknown User•16mo ago

Message Not Public

kian•16mo ago

Yep R2 returns a Marker of foo and a NextMarker of foo so it keeps following NextMarker I'm going to move the files from my R2 bucket onto S3 & run the same request against S3 Specifically it's <NextMarker>Resilience_1.hbk/</NextMarker> (varies between upload jobs but you get the idea) Previously the job uploaded to a folder called sdk-example and it looped on this

    <Marker>sdk-example/</Marker>
    <MaxKeys>1000</MaxKeys>
    <NextMarker>sdk-example/</NextMarker>

    <Marker>sdk-example/</Marker>
    <MaxKeys>1000</MaxKeys>
    <NextMarker>sdk-example/</NextMarker>

It's basically being told 'to check this folder, use this marker' but that marker isn't progressing anything

kian•16mo ago

kian•16mo ago

so NextMarker is Resilience_1.hbk/ so I'll add the marker search param with that value

kian•16mo ago

kian•16mo ago

now I'm stuck forever, since it's just telling me to check where I already am

kian•16mo ago

if I remove the delimiter and the marker, we can see there are files

kian•16mo ago

including other folders after that

kian•16mo ago

if I do the same against S3:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Name>sdk-example-kian</Name>
    <Prefix></Prefix>
    <Marker>Resilience_1.hbk/</Marker>
    <MaxKeys>1000</MaxKeys>
    <Delimiter>/</Delimiter>
    <EncodingType>url</EncodingType>
    <IsTruncated>false</IsTruncated>
</ListBucketResult>

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Name>sdk-example-kian</Name>
    <Prefix></Prefix>
    <Marker>Resilience_1.hbk/</Marker>
    <MaxKeys>1000</MaxKeys>
    <Delimiter>/</Delimiter>
    <EncodingType>url</EncodingType>
    <IsTruncated>false</IsTruncated>
</ListBucketResult>

Let me try and setup a repro that doesn't involve Synology Repro:

/* s3 client setup */

await S3.send(new PutObjectCommand({ Bucket: "sdk-example", Key: "folder/foldertwo/file" }));

console.log(
  await S3.send(
    new ListObjectsCommand({
      Bucket: "sdk-example",
      Delimiter: "/",
      Prefix: "",
      Marker: "folder/",
    })
  )
);

/* s3 client setup */

await S3.send(new PutObjectCommand({ Bucket: "sdk-example", Key: "folder/foldertwo/file" }));

console.log(
  await S3.send(
    new ListObjectsCommand({
      Bucket: "sdk-example",
      Delimiter: "/",
      Prefix: "",
      Marker: "folder/",
    })
  )
);

S3:

  Delimiter: '/',
  IsTruncated: false,
  Marker: 'folder/',
  MaxKeys: 1000,
  Name: 'sdk-example-kian',
  Prefix: ''

  Delimiter: '/',
  IsTruncated: false,
  Marker: 'folder/',
  MaxKeys: 1000,
  Name: 'sdk-example-kian',
  Prefix: ''

R2:

  CommonPrefixes: [ { Prefix: 'folder/' } ],
  Delimiter: '/',
  IsTruncated: false,
  Marker: 'folder/',
  MaxKeys: 1000,
  Name: 'sdk-example',
  NextMarker: 'folder/',
  Prefix: ''

  CommonPrefixes: [ { Prefix: 'folder/' } ],
  Delimiter: '/',
  IsTruncated: false,
  Marker: 'folder/',
  MaxKeys: 1000,
  Name: 'sdk-example',
  NextMarker: 'folder/',
  Prefix: ''

Both ran on empty buckets

kian•16mo ago

kian•16mo ago

Same file/folder path on both To clarify why Marker: "folder/" is used, that's what R2 returns as NextMarker if I omit the marker.

Unknown User•16mo ago

Message Not Public

kian•16mo ago

kian•16mo ago

Let's just say Synology will, uh... ferrissweatmegaspin

Unknown User•16mo ago

Message Not Public

kian•16mo ago

Yes - but we're also within free blobcatbusiness

itsmatteomanf•16mo ago

@kiannh is great, yeah If you want I can keep going 😛 This varies as it’s the base folder name, which defaults to the NAS name with the hbk extension. You can select it.

kian•16mo ago

I should apply as QA /s

James•16mo ago

Glad this was got to the bottom of! Thanks so much for the debug y’all. Would love to use R2 with my synology at some point 👀

Unknown User•16mo ago

Message Not Public

itsmatteomanf•16mo ago

I have my job ready, if you need a test just ping me anytime. It’s a very easy click to test, and even if I need to set-it up again I did a separate token, so I roll and restart easily. It was fun, seeing @kiannh swear at Signatures in S3 CLI 😂

kian•16mo ago

James•16mo ago

Any news here? 👀 Is there a better place I could file this for tracking? Ticket to custesc maybe?

Unknown User•16mo ago

Message Not Public

James•16mo ago

Alrighty, will do 🙂

itsmatteomanf•16mo ago

Should we do one and CC the other into it? 👀

James•16mo ago

can do. I was gonna take a look this weekend, don't have much time this week. Will DM you beforehand in case you've already done one lul

itsmatteomanf•15mo ago

Not sure I’m gonna find the time, as it requires a bunch of info to share and to prepare 😅 @sdnts so, we should CUSTESC it, right? No progress in the meantime?

Unknown User•15mo ago

Message Not Public

itsmatteomanf•15mo ago

Hopefully gonna do that today, finally...

Gaming

Programming

One of the queries that Synology makes