Package org.apache.nutch.util

Class MimeUtil

java.lang.Object
org.apache.nutch.util.MimeUtil
public final class MimeUtil extends Object
This is a facade class to insulate Nutch from its underlying Mime Type substrate library, Apache Tika. Any Mime handling code should be placed in this utility class, and hidden from the Nutch classes that rely on it.

  • Constructor Details

  • Method Details

    • setPoolSize

      public static void setPoolSize(int poolSize)

    • cleanMimeType

      public static String cleanMimeType(String origType)
      Cleans a MimeType name by removing out the actual MimeType, from a string of the form: 
            <primary type>/<sub type> ; < optional params
      Parameters:
      origType - The original mime type string to be cleaned.
      Returns:
      The primary type, and subtype, concatenated, e.g., the actual mime type.

    • autoResolveContentType

      public String autoResolveContentType(String typeName, String url, byte[] data)
      A facade interface to trying all the possible mime type resolution strategies available within Tika. First, the mime type provided in typeName is cleaned, with cleanMimeType(String). Then the cleaned mime type is looked up in the underlying Tika MimeTypes registry, by its cleaned name. If the MimeType is found, then that mime type is used, otherwise URL resolution is used to try and determine the mime type. However, if mime.type.magic is enabled in NutchConfiguration, then mime type magic resolution is used to try and obtain a better-than-the-default approximation of the MimeType.
      Parameters:
      typeName - The original mime type, returned from a ProtocolOutput.
      url - The given @see url, that Nutch was trying to crawl.
      data - The byte data, returned from the crawl, if any.
      Returns:
      The correctly, automatically guessed MimeType name.

    • getMimeType

      public String getMimeType(String url)
      Facade interface to Tika's underlying MimeTypes.getMimeType(String) method.
      Parameters:
      url - A string representation of the document URL to sense the MimeType for.
      Returns:
      An appropriate MimeType, identified from the given Document url in string form.

    • forName

      public String forName(String name)
      A facade interface to Tika's underlying MimeTypes.forName(String) method.
      Parameters:
      name - The name of a valid MimeType in the Tika mime registry.
      Returns:
      The object representation of the MimeType, if it exists, or null otherwise.

    • getMimeType

      public String getMimeType(File f)
      Facade interface to Tika's underlying MimeTypes.getMimeType(File) method.
      Parameters:
      f - The File to sense the MimeType for.
      Returns:
      The MimeType of the given File, or null if it cannot be determined.