XQuery/Link gathering

Motivation

You want to gather the links on a blog page.

Method

We use the doc() function to perform an HTTP GET on a remote web page. If the page is a well formed XML file you can then extract all the unorder list items by adding a ul predicate to the doc function.

This script fetches the blog page and selects the urls in the link section, which reference other blog articles. Each referenced article is fetched and the urls marked as external are selected. The result is returned as XML.

declare namespace q = "http://www.w3.org/1999/xhtml";

<results>
{
let $nav := doc("http://www.advocatehope.org/tech-tidbits/theory-of-the-web-as-one-big-database")//q:ul[@class="portletNavigationTree navTreeLevel0"]
for $href in $nav//@href
let $page := data($href)
let $content := doc($page)//q:div[@id="content"]
for $links in $content//q:a[@title="external-link"]
return
   <link>{ data($links/@href) }</link>
}
</results>

Execute

Version 2

Dropping the intermediate variables allows the structure to be seen more clearly:

declare namespace q = "http://www.w3.org/1999/xhtml";

let $uri := "http://www.advocatehope.org/tech-tidbits/theory-of-the-web-as-one-big-database"
return
<results>
{
       for $page in doc($uri)//q:ul[@class="portletNavigationTree navTreeLevel0"]//@href
       for $link  in doc($page)//q:div[@id="content"]//q:a[@title="external-link"]/@href
       return 
          <link>{data($link)}</link>
}
</results>

Execute

Repository Schemas

Daniel is proposing a standard for supporting the extraction of data such as this from a site. Such a schema would define a view of a set of documents sufficient to allow the extraction above to be based on the schema.

We can go some way towards this with a view schema represented as an ER model, with added implementation-dependent paths.

<model  name="blog-links">
    <type name="url" datatype="string"/>
    <entity name="page"  >
        <attribute name="inner"  max="N"   path="//q:ul[@class='portletNavigationTree navTreeLevel0']//@href"  type="page"/>
        <attribute name="external" max="N"  path="//q:div[@id='content']//q:a[@title='external-link']/@href"  type="url"/>
     </entity>
</model>

This schema can then be used by a generic script link gathering script:

let $start := request:get-parameter("page",())
let $view := request:get-parameter("view",())
let $schema := doc($view)
let $inner := $schema//entity[@name='page']/attribute[@name='inner']/@path 
let $external := $schema//entity[@name='page']/attribute[@name='external']/@path  
return
<results>
{
       for $page in util:eval(concat('doc($start)',$inner))
       for $link in util:eval(concat('doc($page)',$external))
       return      
         <link>{string($link)}</link>
}
</results>

This script now performs the task of link gathering on any site whose page structure can be defined in terms of the schema with appropriate paths.

Execute

Relative and absolute URIs

The previous version works only if the URIs are absolute. A little more work is needed if not:

declare namespace q = "http://www.w3.org/1999/xhtml";

declare variable $start := request:get-parameter("page",());
declare variable $view := request:get-parameter("view",());
declare variable $schema := doc($view);
declare variable $base :=  substring-before($start,local:local-uri($start));

declare function local:local-uri($uri) {
  if (contains($uri,"/"))
  then local:local-uri(substring-after($uri,"/"))
  else $uri
};

declare function local:absolute-uri($url) {
   if (starts-with($url,"http://"))
   then $url
   else concat($base,$url)
};

let $inner := $schema//entity[@name='page']/attribute[@name='inner']/@path 
let $external := $schema//entity[@name='page']/attribute[@name='external']/@path  

let $starturi := local:absolute-uri($start)
return
<results>

{
       for $page in util:eval(concat('doc($starturi)',$inner))
       let $pageuri := local:absolute-uri($page)
       for $link in util:eval(concat('doc($pageuri)',$external))
       return      
         <link>{string($link)}</link>
}
</results>

So with a different schema - same model, different paths:

<model name="site-links">
    <type name="url" datatype="string"/>
    <entity name="page"  >
         <attribute name="inner" max="N" path="//div[@class='nav']//a/@href" type="page"/>
        <attribute name="external" max="N" path="//div[@class='content']//a/@href" type="url"/>
    </entity>
</model>

which is a view schema of this test site

Execute

Virtual Paths

The navigation path is still hard-coded in the script. We would like to write path expressions where the steps are defined in the schema. This path would then be interpreted in the context of the schema.

View Schema

In this example, the test site has been expanded to include a separate index page and some additional components in the view:

<model name="site-links">
    <entity name="externalPage">
        <attribute name="title" path="/head/title"/>
    </entity>
    <entity name="index">
        <attribute name="link" max="N" path="//div[@class='index']//a/@href" type="page"/>
     </entity>
     <entity name="page">
        <attribute name="title" path="//head/title"/>
        <attribute name="inner" max="N" path="//div[@class='nav']//a/@href" type="page"/>
        <attribute name="external" max="N" path="//div[@class='content']//a/@href"
        type="externalPage"/>
         <attribute name="author" min="0" path="//div[@class='content']/span[@class='author']"/>
    </entity>
</model>

Index

Path language

This prototype uses a simple path language.The step -> dereferences a relative or absolute URL. Where a step is recognised as an attribute of the current entity, the associated path expression is used, otherwise the step is executed as XPath. The first step identifies the (entity) type of the initial document.

For example:

index/link/->/title

List the titles of the pages in the index.

import module namespace vp ="http://www.cems.uwe.ac.uk/xmlwiki/vp" at "../Gov/vp.xqm";

let $uri := "http://www.cems.uwe.ac.uk/xmlwiki/Gov/site/index.html"
let $schema := "/db/Wiki/Gov/site3.xml"
return
  <result>
  {vp:process-path($uri,"index/link/->/title",$schema) }
  </result>

Run

index/link/->/author/string(.)

List the authors of the pages referenced in the index.

import module namespace vp ="http://www.cems.uwe.ac.uk/xmlwiki/vp" at "../Gov/vp.xqm";

let $uri := "http://www.cems.uwe.ac.uk/xmlwiki/Gov/site/index.html"
let $schema := "/db/Wiki/Gov/site3.xml"
return
  <result>
  {vp:process-path($uri,"index/link/->/author/string(.)",$schema) }
  </result>

Run

page/inner/->/external

List the url of all distinct external links of all pages referenced by the index page.

import module namespace vp ="http://www.cems.uwe.ac.uk/xmlwiki/vp" at "../Gov/vp.xqm";
declare option exist:serialize "method=xhtml media-type=text/html";

let $uri := "http://www.cems.uwe.ac.uk/xmlwiki/Gov/site/index.html"
let $schema := "/db/Wiki/Gov/site3.xml"
return
  <ul>
    {for $uri in distinct-values(vp:process-path($uri,"index/link/->/external",$schema))
    order by $uri
    return
    <li>
    <a href="{$uri}">{string($uri)}</a>
    </li>
   }
  </ul>

Run

page/inner/->/inner/->/title

List the titles of pages linked to the initial page.

import module namespace vp ="http://www.cems.uwe.ac.uk/xmlwiki/vp" at "../Gov/vp.xqm";

let $uri := "http://www.cems.uwe.ac.uk/xmlwiki/Gov/site/test1.html"
let $schema := "/db/Wiki/Gov/site3.xml"
return
  <result>
  {vp:process-path($uri,"page/inner/->/inner/->/title",$schema) }
  </result>

Run

Script

The core function processes a virtual path in the context of a schema.

declare function vp:process-steps($nodes,$context,$steps,$base,$schema) {
if (empty($steps))
then $nodes
else 
 let $step := $steps[1]
 let $entity := $schema//entity[@name=$context]
 return 
    if ( $step = "->" )
    then 
          let $newnodes :=
               for $node in $nodes 
               return vp:get-doc($node,$base)
           return
             vp:process-steps($newnodes, $context, subsequence($steps,2),$base,$schema)
   else
   if ($entity/attribute[@name=$step])
      then 
      let $attribute :=$entity/attribute[@name=$step]
      let $next := string($schema//entity[@name=$attribute/@type]/@name)
      let $path := string($attribute/@path)
      let $newnodes := 
             for $node in $nodes
             let $newnode := util:eval(concat("$node",$path))
             return $newnode
      return
          vp:process-steps($newnodes, $next, subsequence($steps,2),$base,$schema)
  else 
       let $newnodes := 
             for $node in $nodes
             let $newnode := util:eval(concat("$node/",$step))
             return $newnode
      return
          vp:process-steps($newnodes, $context, subsequence($steps,2),$base,$schema) 
};

Acknowledgments

This example is based on an article by Daniel Bennett [1].