It is currently May 15th, 2024, 2:26 am
View unanswered posts | View active topics
Board index
» themaLeecher
» Suggestions
themaPoster
themaManager
themaRegister
themaCreator
|
Page 1 of 1
|
[ 6 posts ] |
|
Author |
Message |
Pablo01
|
|
Post subject: Filter Duplicates By URL. Posted: December 23rd, 2021, 6:48 am |
|
Joined: November 23rd, 2014, 6:31 am Posts: 276
|
Hi,
I am looking for a way/option to prevent TL to add messages that have identical URLs.
I am scraping some websites for posts and I have thousands of duplicate posts because the subjects have been slightly changed.
I would like a way to block for messages being added if identical URL is already in TL.
Alternatively I have to figure out a way to export all messages and filter out dupliates and then re-add into TL - but I dont know really how to do this.
|
|
|
|
Freddy
|
Post subject: Re: Filter Duplicates By URL Posted: December 23rd, 2021, 7:43 am |
|
Site Admin |
|
Joined: March 10th, 2011, 11:14 pm Posts: 12660 Location: Earth
|
Hi,
subjects definitely don't change the same day multiple times. No one is doing that. Checking latest posts by subjects is enough.
When leeching pages it only checks the latest posts. If someone will change subject on very old post it won't matter, themaLeecher won't leech that anyway since it won't be in "Page 1" anymore.
Will add the duplicate URL filter.
_________________ themaPoster | themaCreator | themaManager | themaLeecher | themaRegister
|
|
|
|
Pablo01
|
Post subject: Re: Filter Duplicates By URL Posted: December 23rd, 2021, 8:01 am |
|
Joined: November 23rd, 2014, 6:31 am Posts: 276
|
Thanks Freddy,
I agree nobody change the subject that much or often.
The problem is more how I leech these messages - I have setup the subject labelling change and therefor the messages are multiple times in TL for me.
So basically its because of the way I get the msgs - not the way websites label subjects - and since I may be changing the subject multiple times it will be very nice with the url filter as a duplicate protection.
|
|
|
|
Freddy
|
Post subject: Re: Filter Duplicates By URL. Posted: December 23rd, 2021, 8:52 am |
|
Site Admin |
|
Joined: March 10th, 2011, 11:14 pm Posts: 12660 Location: Earth
|
You can change the subject as many times as needed inside the program. That does not affect duplicate checking by subject. The program saves original subject internally and it's used for comparing (not the subject which you see in the program).
Will add the duplicate URL filter.
Edit: 4.50: * Added "Duplicate URLs" filter in messages "Search and filter" window.
_________________ themaPoster | themaCreator | themaManager | themaLeecher | themaRegister
|
|
|
|
Pablo01
|
Post subject: Re: Filter Duplicates By URL. Posted: January 2nd, 2022, 8:19 am |
|
Joined: November 23rd, 2014, 6:31 am Posts: 276
|
Thanks Freddy,
Now I can filter them out after they are leeched - but as I understand this does not prevent TL from leeching messages with same URL as messages already existing in TL.
The problem for me is that there is a Video Price in the subject - and these sellers sometimes change the video price and then TL picks up the message again and then its a duplicate - just not by subject - but by URL.
|
|
|
|
Freddy
|
Post subject: Re: Filter Duplicates By URL. Posted: January 3rd, 2022, 8:22 am |
|
Site Admin |
|
Joined: March 10th, 2011, 11:14 pm Posts: 12660 Location: Earth
|
|
|
|
|
Page 1 of 1
|
[ 6 posts ] |
|
Board index
» themaLeecher
» Suggestions
themaPoster
themaManager
themaRegister
themaCreator
Who is online
Users browsing this forum: No registered users and 33 guests
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot post attachments in this forum
|