Which approach to display a preview of urls?

TL;DR I am currently trying to implement an URL preview for a chat feature (like on Whatsapp, Line, Telegram, etc) on mobile.

Stack: Mobile clients (iOS/Android app) and an API server (Django, MariaDB).

Flow: Users send to each other messages containing links to any website.

Goal: I want to display a preview of the URL using only meta tags (either Open Graph or basic meta tags).

I see 2 different approaches:

  • Server-driven: The clients (iOS/Android) ask the API for a preview of a link. The API server will fetch the URL directly if not in cache, then cache the result and serve it to the clients.
  • Client-driven: The clients directly fetch the link and display the preview.

Is there any other approach? Which one is the standard? Which one should be avoided?

For Messenger and Whatsapp, I believe the client requests Facebook API for a preview as Facebook has tons of OpenGraph preview data of almost all websites. But what about the others?

Urls no responden – Django

Tengo un problema con mis urls en Django, aquí mi urls.py:

from .views import (     CouponListView, CouponDetailView, buy_coupon, UserCouponListView, CouponOfResaleCreateView,     CouponOfResaleListView, )   from django.urls import path, include  coupons_patterns = ([     path('', CouponListView.as_view(), name = 'list'),     path('<slug:slug>/', UserCouponListView.as_view(), name = 'user'),     path('<int:pk>/<slug:slug>/', CouponDetailView.as_view(), name = 'detail'),     path('coupon/<int:pk>/buy-<slug:slug>/', buy_coupon, name = 'buy-coupon'),      path('resale/', include([         path('', CouponOfResaleListView.as_view(), name = 'resale_list'),         path('create/<int:pk>/<slug:slug>/', CouponOfResaleCreateView.as_view(), name = 'create'),     ])),  ], 'coupons') 

Las urls que no funcionan o que no responden son estas:

path('resale/', include([         path('', CouponOfResaleListView.as_view(), name = 'resale_list'),         path('create/<int:pk>/<slug:slug>/', CouponOfResaleCreateView.as_view(), name = 'create'), ])), 

No me sale error solo no responden, es decir, si yo quiero acceder a una de las urls que mostré anteriormente, por ejemplo a la resale_list en ves de mostrarme lo que debería mostrarme, me muestra el resultado de acceder a la url user.

En la url del navegador esta la url correctamente pero me muestra el HTML de otra url, que en este caso es la url user.

La forma de solucionarlo es mover esa porción de urls arriba, de la siguiente manera:

coupons_patterns = ([     path('resale/', include([         path('', CouponOfResaleListView.as_view(), name = 'resale_list'),         path('create/<int:pk>/<slug:slug>/', CouponOfResaleCreateView.as_view(), name = 'create'),     ])),      path('', CouponListView.as_view(), name = 'list'),     path('<int:pk>/<slug:slug>/', CouponDetailView.as_view(), name = 'detail'),     path('coupon/<int:pk>/buy-<slug:slug>/', buy_coupon, name = 'buy-coupon'),     path('<slug:slug>/', UserCouponListView.as_view(), name = 'user'),  ], 'coupons') 

Otra solución es mover la url user abajo de dicha porción de urls, de la siguiente manera:

coupons_patterns = ([     path('', CouponListView.as_view(), name = 'list'),     path('<int:pk>/<slug:slug>/', CouponDetailView.as_view(), name = 'detail'),     path('coupon/<int:pk>/buy-<slug:slug>/', buy_coupon, name = 'buy-coupon'),      path('resale/', include([         path('', CouponOfResaleListView.as_view(), name = 'resale_list'),         path('create/<int:pk>/<slug:slug>/', CouponOfResaleCreateView.as_view(), name = 'create'),     ])),      path('<slug:slug>/', UserCouponListView.as_view(), name = 'user'),  ], 'coupons') 

Pero ¿Por que pasa esto?, ¿Porque dependiendo el orden de las urls hace que suceda dicho problema?.

¿Alguna solución?, ya que no me gusta tener mis urls en ese orden para que funcionen.

Ecommerce SEO and URLs

Hey there

New to SEO. Just wondering if having /collections/ in the URL hurts search results or if it's in any way advantageous to have it removed. Ultimately it's not necessary to have, but would removing it on an existing site (using shopify) be more trouble than it's worth?

Thank you!

Find website URLs that begin with http:// or https://

This is a problem from Automate the Boring Stuff from “Pattern Matching with Regular Expression”. I am a beginner in Python and the purpose of this program is to match urls which start with http:// or https://, with some urls having an optional www. after https://

How can this code be improved?

#! /usr/bin/python3 #websiteURL.py - Find website URLs that begin with http:// or https://  import pyperclip, re  #protocol Regex - checks for http:// or https:// protocolRegex = re.compile(r'''     https?://           #match http:// or https://     (?:w{3}\.)?         #www-dot     [a-zA-Z0-9_-]+      #domin name     \.                  #dot     [a-zA-Z]{2,3}       #extension     ''', re.VERBOSE)  text = str(pyperclip.paste()) #copying data from document to clipboard and converting it into a string matches = [] #stores all matches in this list  for website in protocolRegex.findall(text): #finding website from the string text     matches.append(website)  if len(matches) > 0:     pyperclip.copy('\n'.join(map(str, matches))) #copying result to clipboard after adding newline after each match     print('Copied to clipboard:')     print('\n'.join(matches)) else:     print('No website found') 

Running code:

chmod +x websiteURL.py ./websiteURL.py 

How to deal with long URLs in comments that exceed the 80 characters limit?

I generally try to adhere to the 80 character limit for each line of code to satisfy my colleagues, to satisfy the linting tools, and to maintain an image of professionalism when random people look at my code.

However, sometimes I have URLs that I wish to embed in the comments, and some of these URLs are very long (exceeding 80 characters). Exaggerated example: https://github.com/django/django/blob/42b9a23267f14be39b9b00958e18d5746783208e/django/contrib/auth/management/commands/changepassword.py#L17-L21

How do professional software engineers deal with this problem? Do they make an exception to the 80 characters rule for long URLs?

Scrapy doesnot yield website urls of DNS Lookup failed websites

I have a list of urls that gets redirected to another url in a text file. I want to get all redirected urls.So I did a spider that opens url from a text file. Now I get some error for few as “DNS look up failed” or “No route”. I checked those urls directly on browser and found those url get “ip address not found error”. However I want scrapy to every redirected urls regardless of the error. Any soulutions to achieve this?

Here is the spider I ran

import scrapy  class AwesomeSpiderSpider(scrapy.Spider):     name = 'web_uk' # opening the list of urls that gets redirected.     f = open("urls.txt")     start_urls = [url.strip() for url in f.readlines()]     f.close()       def parse(self,response):          item = {}         item['Web Address'] = response.url         yield item 

Here is the output

2019-07-04 03:02:03 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://gatesidedevelopments.com/> (referer: None) ['cached'] 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://honka.com/gb/en/> {'Web Address': 'https://honka.com/gb/en/'} 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.dapconstruction.co.uk/> {'Web Address': 'https://www.dapconstruction.co.uk/'} 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.virtueprojects.com> {'Web Address': 'http://www.virtueprojects.com'} 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://waynemoore.com/> {'Web Address': 'https://waynemoore.com/'} 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.avenuenorth.co.uk/> {'Web Address': 'http://www.avenuenorth.co.uk/'} 2019-07-04 03:02:03 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.galkivconstruction.co.uk/robots.txt> (failed 1 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk. 2019-07-04 03:02:03 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.mic.uk.com> {'Web Address': 'http://www.mic.uk.com'} 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.vlconstruction.co.uk/> {'Web Address': 'https://www.vlconstruction.co.uk/'} 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.whitehalloflondon.co.uk> {'Web Address': 'http://www.whitehalloflondon.co.uk'} 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.vandthomes.com> {'Web Address': 'http://www.vandthomes.com'} 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.atlanticdwellings.com/> {'Web Address': 'https://www.atlanticdwellings.com/'} 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.rgfhomeimprovements.co.uk> {'Web Address': 'http://www.rgfhomeimprovements.co.uk'} 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://leonoc.co.uk/> {'Web Address': 'https://leonoc.co.uk/'} 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 http://www.home-refurbishments.co.uk/> {'Web Address': 'http://www.home-refurbishments.co.uk/'} 2019-07-04 03:02:03 [scrapy.core.scraper] DEBUG: Scraped from <200 https://gatesidedevelopments.com/> {'Web Address': 'https://gatesidedevelopments.com/'} 2019-07-04 03:02:07 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.rrawlins.co.uk/robots.txt> (failed 1 times): An error occurred while connecting: 113: No route to host. 2019-07-04 03:02:07 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36 2019-07-04 03:02:11 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.galkivconstruction.co.uk/robots.txt> (failed 2 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk. 2019-07-04 03:02:11 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.45 Safari/537.36 2019-07-04 03:02:12 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.rrawlins.co.uk/robots.txt> (failed 2 times): An error occurred while connecting: 113: No route to host. 2019-07-04 03:02:12 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 2019-07-04 03:02:17 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.galkivconstruction.co.uk/robots.txt> (failed 3 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk. 2019-07-04 03:02:17 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET http://www.galkivconstruction.co.uk/robots.txt>: DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk. Traceback (most recent call last):   File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks     result = result.throwExceptionIntoGenerator(g)   File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator     return g.throw(self.type, self.value, self.tb)   File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request     defer.returnValue((yield download_func(request=request,spider=spider)))   File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks     current.result = callback(current.result, *args, **kw)   File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 975, in startConnectionAttempts     "no results for hostname lookup: {}".format(self._hostStr) DNSLookupError: DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk. 2019-07-04 03:02:17 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36 2019-07-04 03:02:19 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.rrawlins.co.uk/robots.txt> (failed 3 times): An error occurred while connecting: 113: No route to host. 2019-07-04 03:02:19 [scrapy.downloadermiddlewares.robotstxt] ERROR: Error downloading <GET http://www.rrawlins.co.uk/robots.txt>: An error occurred while connecting: 113: No route to host. Traceback (most recent call last):   File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request     defer.returnValue((yield download_func(request=request,spider=spider))) ConnectError: An error occurred while connecting: 113: No route to host. 2019-07-04 03:02:19 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36 2019-07-04 03:02:22 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.rrawlins.co.uk/> (failed 1 times): An error occurred while connecting: 113: No route to host. 2019-07-04 03:02:22 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.79 Safari/537.36 2019-07-04 03:02:23 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.galkivconstruction.co.uk> (failed 1 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk. 2019-07-04 03:02:23 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.43 Safari/537.31 2019-07-04 03:02:29 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.galkivconstruction.co.uk> (failed 2 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk. 2019-07-04 03:02:29 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 2019-07-04 03:02:29 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET http://www.rrawlins.co.uk/> (failed 2 times): An error occurred while connecting: 113: No route to host. 2019-07-04 03:02:29 [scrapy_user_agents.middlewares] DEBUG: Assigned User-Agent Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36 2019-07-04 03:02:34 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.galkivconstruction.co.uk> (failed 3 times): DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk. 2019-07-04 03:02:34 [scrapy.core.scraper] ERROR: Error downloading <GET http://www.galkivconstruction.co.uk> Traceback (most recent call last):   File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks     result = result.throwExceptionIntoGenerator(g)   File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator     return g.throw(self.type, self.value, self.tb)   File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request     defer.returnValue((yield download_func(request=request,spider=spider)))   File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks     current.result = callback(current.result, *args, **kw)   File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/twisted/internet/endpoints.py", line 975, in startConnectionAttempts     "no results for hostname lookup: {}".format(self._hostStr) DNSLookupError: DNS lookup failed: no results for hostname lookup: www.galkivconstruction.co.uk. 2019-07-04 03:02:36 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET http://www.rrawlins.co.uk/> (failed 3 times): An error occurred while connecting: 113: No route to host. 2019-07-04 03:02:36 [scrapy.core.scraper] ERROR: Error downloading <GET http://www.rrawlins.co.uk/> Traceback (most recent call last):   File "/home/ubuntu/scrapy/local/lib/python2.7/site-packages/scrapy/core/downloader/middleware.py", line 43, in process_request     defer.returnValue((yield download_func(request=request,spider=spider))) ConnectError: An error occurred while connecting: 113: No route to host. 2019-07-04 03:02:36 [scrapy.core.engine] INFO: Closing spider (finished) 

When you scrape for your own taget urls

When you scrape for your own taget url how many keywords do you need?

I tried to scrape my own through gsa and post but i dont get any backlinks.

I use 1 gmail account and 10 private poxies with also public poxies. 

I am trying to create swedish backlinks mostly articles but i cant do it.  And also i cant create web 2.0 and.

Also can i get free swedish articles through search ung online from gsa ranker software