Tuesday, September 30, 2008

Progress - News Beta Testing...

by Phil 'iwonder' Guerra
--- (Mission, KS). While testing I found some Atom feeds failing to download. It's strange, but I did manage to get at least one to download and display, but others failed. I took a look at the conversion xsl files in the RssToolKit.

I actually found a line in the AtomToRss20.xsl that seemed to be the culprit, moreso than the date cnversion, although there is an issue with that too. The basic reason why it wouldn't load was becasue the conversion xsl has an invalid directive:

<xml:output mode="xml" encoding="UTF-8"/>

This was probably throwing an error during the attempt to convert it or download it. I put the xsl into my xsl editor, with an input Atom xml file, and it choked, with an error: The namespace prefix is not allowed to start with the reserved string "xml". So, I replaced that line with this one:

<xsl:output method="xml" encoding="UTF-8"/>

Then, it began to work. The date conversion routine had issues too, with not producing a proper formatted date. I corrected that issue, too. I'll send the updated xsl file to Peter, as soon as I can go through the other files in the RssToolKit.

Strange thing, though I don't know why one of the feeds I tested, actually did work.?!? I'll have to go back to that one, to check it out more closely.

Now, after looking at the conversion xsl again, I noticed some other issues with it. For example, the title links to a comment area, and not the actual article. More work needs to be done.

Friday, September 26, 2008

DNN Make Feed Display on All Pages

by Phil 'iwonder' Guerra
--- (Mission, KS). A common question about the News Module, or any DNN core module for that matter, is how to make it show up on all portal web pages. It's pretty simple, but can be easily overlooked. Here's the simple solution, most modules have an option that give an administrator the option to display it on all pages.

In the news module, just go to the module settings, expand the Advanced Settings area and check the box next to the settings label:
Display Module On All Pages?

This will display the module in all of the pages. I use it to display a Headline ticker in the top pane on all of the pages.

Most core DNN modules work the same way.

Wednesday, September 24, 2008

Localized RSS Date Time...

by Phil 'iwonder' Guerra
--- (Mission, KS). Not sure why folks want to change the pubDate format given in a newsfeed. I have nothing but trouble when I try to base anything on pubDate. I can understand the desire to show feeds with localized date/time, but seems like a whole lot of trouble to go through. My experience with most feeds generated is that they don't even reflect correct information in about 80% of the usage I've seen. The discrepency usually can be attributed to developers not understanding how to correctly use date/time fields and convert them, and the further complication of date/times being formatted in databases based on the server date/time setup, which may not be correctly setup or maintained.

It's really puzzling to me how RSS specs decided to use the RFC822 date/time format in the first place. This was a standard developed for first gen email text messages. Peter's correct in that it was an American originated spec, but it did have a group of international folks working on it, which is why the format doesn't even match American date formats either. Now to really confuse matters, the RFC822 spec is really obsolete, the newer RFC2822 is the defacto standard. So, why in the world do we not use it? Well, it isn't all that much better for this type of usage, folks.

Perhaps a better standard would be ISO8601 (see a Wiki entry here. If you still are not totally confused, and want to press further, I suppose you are just bent that way, and since you are using MS prodcuts with DotNetNuke, you could prolly find a way to do this in the News module, since it's rewritting the incoming news source anyway. However, that brings us back to the lack of folks using any standard to start. Any code fix requires some kind of 'rule' to fallback on, and with many news feeds, there just isn't any reliability. Also, what do you suggest is used for the default language? Is it the portal setting for preferred language, the host server setting for language, or the end-user's machine language setting? Those questions have never been answered in the core DotNetNuke code. I know because, I tried to get folks to make a decision on it, and never got any DNN Core Team members to attach enough significance to it to make it happen. It's still not resolved in the DNN.

In fact, to be fair, it's not really resolved in Microsoft's world either. Yes, I know about the Olson timezone, tz database in use on Unix, Linux, and other implementations. I am a member of the tz mail list, and over the years, even that solution goes through incredible changes in year, almost daily. I've come to the conclusion that until the world comes under one umbrella, there is not going to be a solution to please everyone. Although, if you need me to vote, I'd say hitch your wagon to the tz database if you can, at least it has provision for being used for historical context, supported by users throughout the globe, not just in Redmond.

Now, you can change the format using XSL, but it's a real chore to provide a solution that's universal and fully globalized. Most solutions are going to be local solutions as far as I can tell. I've done a bit of that stuff, but reverted back to not caring about it because most feeds don't always give you pubDate in the rfc822 format, and trying to figure out universal routines to deal with all the variances of gloablized date/time formats is difficult to say the least. So, I just take it as it is - much like the content in a feed.

I don't even like to validate the pubDate format because I'd rather worry about getting the content than stressing over whether the pubDate is valid and miss displaying the content. Maybe I'm funny that way, but my usage of feeds is not based on pubDate, and if I choose to cache, I think the better approach would be to simulate a ttl element based on the date/time I pulled the feed originally, if one is not available. I'd use whatever is there, if easily parsed, so I can display the pubDate as given, and still have control over timing for feed retrieval and display. Just my thoughts...

Tuesday, September 23, 2008

RSS XSL Stripping HTML

by Phil 'iwonder' Guerra

I get a fair amount of questions about how to handle transformations, either related to custom XML data islands or RSS newsfeeds. One question that seems to be near the top of the list is "How can I get rid of all HTML in the description of an RSS newsfeed?" Well, the answer involves a bit of study of the source, and then, a bit of tweaking, and testing of a custom XSL.

Searching the web, I located several examples of XSL functions that I could tweak to provide a flexible solution that integrates well into the DotNetNuke framework, either with the News module or the XML/XSL module. I found the code that worked the best for me at this URL:

http://code.techinterviews.com/xslt-to-strip-html/26

Stripping HTML Template

<xsl:template name="strip_HTML">
<xsl:param name="value"/>
<xsl:choose>
<xsl:when test="contains($value,'&lt;')">
<xsl:value-of select="substring-before($value,'&lt;')" disable-output-escaping="yes"/>
<xsl:choose>
<xsl:when test="contains(substring-after($value,'&lt;'),'&gt;')">
<xsl:call-template name="strip_HTML">
<xsl:with-param name="value"><xsl:value-of select="substring-after($value,'&gt;')"/></xsl:with-param>
</xsl:call-template>
</xsl:when>
<xsl:otherwise>
</xsl:otherwise>
</xsl:choose>
</xsl:when>
<xsl:otherwise>
<xsl:value-of select="$value" disable-output-escaping="yes"/>
</xsl:otherwise>
</xsl:choose>
</xsl:template>


Using my basic XSL, I added the template for 'stripping-HTML' and tweaked it until it functioned as I needed. Now, it's just a matter of calling the template when I want to use it. In this case, I want to strip the HTML from the <description> element, so the call to the template is done this way:


<xsl:call-template name="strip_HTML">
<xsl:with-param name="value" select="description" />
</xsl:call-template>


Friday, September 19, 2008

DNN News Failing? Relax...

by Phil 'iwonder' Guerra

Woe is us, huh?!? The DNN News module is failing on many newsfeeds sources that worked without issue on the previous version. Why? Well, the new module places some very tight constraints on the incoming newsfeed used. They must be able to be successfully transformed into a valid RSS v2.0 to be displayed using the News module v04.00.00. Simply, a newsfeed will fail if the elements do not pass validation specified in the module's XSD file.

The problem is most newsfeeds do not care about conforming to any RSS spec, their developers do not understand what is needed, or the users creating them do not understand what to use in the placeholders to which they add content. Anyway you slice it, there is a problem that is causing many users to shy away from the new module, or move back to the previous version of the News module. There's even a thread to report a feed that does not work here.

Now, the issue is a hot topic on the forums, especially visible within the first few days of the modules' release. In the process of these discussion some folks found a workaround that takes the steps of relaxing those constraints, by modifying the modules' XSD file.

You can read about the issue here on the forums, or take a look at another's view of the situation here, which includes some detail on how to fix it. I've already commented on the issue, but am adding my fix, which is basically the same as Craig's. He beat me to posting it. No matter, because the point is folks needed a fix.

Since, I knew about the XSD issue, but didn't yet have the source, I couldn't fix the problem. The fix involves modifying the Rss20.xsd file located in the \Website\DesktopModules\News\RssToolkit\Resources folder. There you will find the description of the elements used to validate a feed which eventually gets transformed from its' source format, Atom, RDF, RSSv0.91, RSSv1.0 to RSSv2.0.

The constraints used to descibe some elements can be lifted entirely, or you could manage to ease them back a tad, depending on your capability and needs. I chose to relax them entirely for the elements using these custom types:

- tEmailAddress
- tRfc822FormatDate

<xs:simpleType name="tEmailAddress">
<xs:annotation>
<xs:documentation>Not Using the regexp definiton of E-Mail Address </xs:documentation>
</xs:annotation>
<xs:restriction base="xs:string">
</xs:restriction>
</xs:simpleType>
<xs:simpleType name="tRfc822FormatDate">
<xs:annotation>
<xs:documentation>A date-time displayed in RFC-822 format.</xs:documentation>
<xs:documentation>Not Using the regexp definiton of rfc-822 date </xs:documentation>
</xs:annotation>
<xs:restriction base="xs:string">
</xs:restriction>
</xs:simpleType>

Making those changes, recompiling the module, and placing the resulting DotNetNuke.Modules.News.dll into the \Website\Bin folder fixed many of my issues. There are still some odd ones floating about, like constraints on image size, but this first set of mods, gives me the most benefit.

Now, is this the way to fix it overall? I don't think it is a 'best practice' by any means. I'm testing it at this point, so you can follow this example and test it yourself, or wait for the 'official' fix. My plan is to provide validation at the time of adding a feed, rather than validating after the feed is fetched. Also, I think it better to provide a gracefull correction to these issues by coding a way to provide a usable alternative element.

I'll post more details later. BTW - Thanks to Craig Hubbard for adding the information in the forums about how he fixed his issues. I was floundering about making the XSD changes, and didn't realize I had to recompile the project for it to pick up the changes.

Thursday, September 18, 2008

Building the DNN News Module With Visual Web Developer 2008

by Phil 'iwonder' Guerra

With all of the new tools available from Microsoft, and so many swift changes in technology, it's really difficult to keep up. I took some time off from DNN development to focus on HPUX System Administration, which is another topic. In the course of that time, Microsoft released the many varieties of Vista, VS2008, SQL2008, SQL2008Express, and Visual Web Developer 2008 among others. Phew, what's a self employed single consultant to do? Can't go out and buy all of the bells and whistles. So, I'm setting up shop again, using nothing but freebies. Of course, many who know me, would say what's different? Nothing, exactly, I did get the more pricey tools as part of payment for consulting work, but I really want to go about just using tools that others have readily available to them. It makes teaching folks how to take care of things themselves much easier.

Anyways, part of the ongoing discussion in the DNN News forums is about how to fix the strict validation of incoming newsfeeds. The validation is done using a xsd that specifies the restrictions of fields used by the module. While you can change the xsd to relax the constraints, you will need to recompile the News module for your changes to take affect. That's where this tidbit comes in...

The primary DNN project comes in 2 flavors at the moment, a vs2005 and vs2008 solution. The DNN News module was built using either vs2008 or vwd2008. So, if you want to make any changes to the module, you need to use one of those tools. In my case, I'm using VWD2008. Since, I'm not really an expert on it, I asked for some help on the forums, which got some pretty terse response, still gave me enough to 'remember' that the missing ingredient in all of my troubles was just to add the missing references in the project.

If you are using MS VWD 2008, it is possible to make your own changes to the News module. Open up the project in VWD2008, and add the references for the following dlls located in the folder where you put dnn \Website\Bin\:

DoteNetNuke.dll
DotnNetNuke.WebUtility.dll
Microsoft.ApplicationBlocks.Data.dll

Then, try a quick rebuild before making any changes. If your setup is acceptable, the project will rebuild successfully, then the module will be ready for more changes. If you have errors, there may be other references to add, but those are the only ones I needed to add.

After you make your changes, just replace the resulting DotNetNuke.Modules.News.dll in your Website\DesktopModules\News\bin folder (or wherever your source is located), into the \Website\Bin folder.

The easiest fix is to relax the constraints that are causing folks so much trouble with their existing feed sources. Thats the next topic...

Saturday, September 13, 2008

DNN News Module Too Strict?

by Phil 'iwonder' Guerra

One thing that is difficult to manage is the aggregation of feeds without standards. Unfortunately, the strict enforcement of RSS v2.0 implementation is more for the aggregation rather than just the display of a single feed. At this point, I believe the DNN News module is being too strict, as most newsfeed sources will never all comply with the specs for any version of the RSS or Atom specs. One of the more common errors is in the display of date of publication, which is not even a standard element between RSS v2.0, RSS v1.0 (RDF), and Atom feeds. Also, the date formatting is totally out of whack for for a lot of feeds, even when somethings gives GMT as its' base, you cannot rely on the develper actually giving you that correctly.

Other troublesome elements are those that are supposed to include and email in the content, but only provide a name. This is a common error emitted for a lot of popular newsfeeds, but one easily avoided with including the Dublin Core namespace in a feed, and using one of the 3 included metadata elements that do not require email:

- creator
- publisher
- contributor

There are other DC elements that are useful, as well. In fact many newsfeeds incorporate them as extensions to whatever version RSS they emit.

The other option is for the DNN module to include the Dublin Core namespace and use one of the above elements in place of the RSS v2.0 author element when an email format fails, and always provide the associated DC element with the email stripped when one is present. In this way the feed validates and can be used without penalty.

With the apps I write for desktop, I just take this information without trying to validate it or use it for anything other than displaying it as given. If I can't make sense of a date element in a feed, I find it easier to fashion a date for sorting, or caching, which I take from the time I first bring in the element. It's not going to make a whole lot of difference in the presentation anyway IMHO. The main usage for me of having a pubDate is for sorting, and accuracy in attribution of the news source.

It's mostly a matter of lack of attention by developers writing the feed generators. We see it in the DNN modules as well. The whole point of specs is to keep folks from having to worry about issue with code that is supposed to be compliant. Of course, that's not what we see in the real information world. If auto makers disregarded safety specifications as much as developers, we'ld have pulled the offenders off the market. Unfortunately, there is no similar enforcement in our IS world.

I think going into this update, most didn't realize the number of newsources that were not compliant. If something says it's RSSv2.0 then it should be expected to be a valid feed. There's the trouble, even larger news providers don't offer compliant RSS, so it's something the industry just avoids by coding around it. In the end, that's probably what will have to be done in the module, because aggregation should not cause aggrevation.

Wednesday, September 10, 2008

DNN News Module - Headlines.xsl Ticker Available

Two XSL files are included that allow vertical or horizontal scrolling of newsfeeds items in the new DNN News module v04.00.00. After playing around with the new News module, I found a slight implementation issue with using the Ticker.xsl. When I choose to use this xsl, I don't get the expected results, which is seeing a display of the news headlines scroll across my module 1 at a time. Instead the xsl displays all of the items, and scrolls them all across the module container. If you have 10 or more items, this kind of implementation is really distracting, as all of the items display and are scrolled.

Now, that is not the implementation I've seen with other similar web usage. So, I created an alternative xsl that can be dropped into the modules transformation folder and used to provide the display of News Headlines scrolling across the module container. This is mostly useful for the Top container area, but suppose you could put it elsewhere - I use it in the Top area.

The settings for 'Show Items Date', and 'Show Items Detail' are not used, as this xsl is desigened just to display the headlines (title element), which I think is the more oft used implementation.

The other thing about this new xsl is that it will support the "ItemsToShow" parameter, but keep in mind that if you are agregating a number of feeds, the count is for the total number of items to show, not the number of items to show per feed.

I placed this HeadlinesTicker.xsl in the same folder as the other default News xsl files

\Website\DesktopModules\News\Transformations

I'm guessing that you could upload it, and choose it for using for your module, as it does transform RSSv2.0 feeds, which is the only custom xsl types that I've seen actually work with the new module. Copy and paste the code, save it to a file using your favorite bare bones text editor. Then, place it our upload it. Be sure to test it out first, though to ensure it does what you think it should.

Click on the blog Title to view the DNN News Forum post where I give the actual code. Sorry, can't provide it here.

Monday, September 8, 2008

DNN News Module v04.00.00 Released

The new release of the DotNetNuke News module brings some nice new features to the module. Peter Donker blogs about it on his DNN Blog. You'll need at least DNN v04.06.02 to integrate into your DNN website. Also, the new module has some gotchas, so visit the DNN News Forum for a review of the discussions.

Be aware that the new module is very strict in its' validating a newsfeed source, so don't expect all of your existing newsfeeds to work straight away. The issues causing folks so much trouble are mostly due to the lack of newsfeed providers to emit valid RSS feeds. The most troublesome aspect of all of all is that even certain DNN Core modules suffer this same condition.

Most feeds that are being reported as failing are due to validation failing on the newsfeed item elements that do not provide rfc822 dates or valid email information for elements that require them.

I've been using the module for a bit now, and will blog more about it later. For now, download it, install it in a test environment, and put it to the test. It's more advanced than the previous version, which only used a news source link to make a web request and transform it using xsl. The new module is the second DNN attempt at news aggregation (the FeedExplorer was the first). It works reasonably well, but newbies and some 'oldies' with little experience with RSS and newsfeeds will need help in using it out of the box. The strict RSS validation is rough going.

Also, new is the ability to use passed parameters to control some of the display of a newsfeed. You can control the number of elements and choose to display or not display the item date and details.

There are some included XSL files which allow scrolling and default look and feel. The basic approach is to provide a newsfeed display much like many new browsers are doing, but allowing several differnt types of newsfeeds, Atom, RSSv0.91, RSSv1.0 (RDF), and RSSv2.0 to be used without worrying the details. Of course, the implementation still needs some work, but the overall direction of the DNN News module is finally getting an update.

More on the module later. I'll have some tips, and workarounds posted for you.