Reading Avro into spark using spark-avro -

-

i'm not being able read spark files using spark-avro library. here steps took:

  • got jar from: http://mvnrepository.com/artifact/com.databricks/spark-avro_2.10/0.1
  • invoked spark-shell using spark-shell --jars avro/spark-avro_2.10-0.1.jar

  • executed commands given in git readme:

    import com.databricks.spark.avro._ import org.apache.spark.sql.sqlcontext val sqlcontext = new sqlcontext(sc) val episodes = sqlcontext.avrofile("episodes.avro")

  • the action sqlcontext.avrofile("episodes.avro") fails following error:

    scala> val episodes = sqlcontext.avrofile("episodes.avro") java.lang.incompatibleclasschangeerror: class com.databricks.spark.avro.avrorelation has interface org.apache.spark.sql.sources.tablescan super class @ java.lang.classloader.defineclass1(native method) @ java.lang.classloader.defineclass(classloader.java:760) @ java.security.secureclassloader.defineclass(secureclassloader.java:142) @ java.net.urlclassloader.defineclass(urlclassloader.java:467) @ java.net.urlclassloader.access$100(urlclassloader.java:73) @ java.net.urlclassloader$1.run(urlclassloader.java:368)

my bad. readme says:

versions  spark changed how reads / writes data in 1.4, please use correct version of dedicated spark version  1.3 -> 1.0.0  1.4+ -> 1.1.0-snapshot

i used spark:1.3.1 , spark-avro: 1.1.0. when used spark-avro: 1.0.0, worked.


Comments

Post a Comment

Popular posts from this blog

yii2 - Yii 2 Running a Cron in the basic template -

-
i have seen instructions on running cron in advanced template, cant figure out how on basic template. have following controller in basic controllers folder. <?php namespace app\controllers; use yii\console\controller; /** * cron controller */ class croncontroller extends controller { public function actionindex() { echo "cron service runnning"; } public function actionmail($to) { echo "sending mail " . $to; } } i have navigated root of application , tried these comands yii cron php yii cron im getting unknown comand "cron" i know old question failed find actual answer this. if @ code yii exec file you'll notice requires /common/config/aliases.php file console/config/main.php. can naturally change these or copy on these advanced template. hope solves problem wanting same thing.
Read more

asp.net - 'System.Web.HttpContext' does not contain a definition for 'GetOwinContext' Mystery -

-
i realize question might seem trivial some, it's these types of things find myself fighting quite bit , want make sense of despite seeming losing battle in .net (for me anyway). so, if following: using system.web; ... applicationuser user = system.web.httpcontext.getowincontext().getusermanager<applicationusermanager>().findbyid(system.web.httpcontext.user.identity.getuserid()); that produces error in title , red getowincontext() , error cannot resolve symbol 'getowincontext()' however, if following (remove system.web in front of httpcontext ), works expected (or @ least no errors): using system.web; ... applicationuser user = httpcontext.getowincontext().getusermanager<applicationusermanager>().findbyid(system.web.httpcontext.user.identity.getuserid()); however, if (same line that's working using system.web commented out): //using system.web; ... applicationuser user = httpcontext.getowincontext().getusermana...
Read more

mercurial graft feature, can it copy? -

-
there new feature in mercurial 3, called 'graft' (graft local). job of moving change-set different branch. is there way "copy" change-set branch? i interested in in moving stuff qa branch production branch, still need code reside in qa. want copy. found , have tried few things, maybe it'll become clear more tinkering , reading... does have better way of 'cherry picking' change-sets move cross branch (or trunk)??? graft copies changesets, doesn't move them. hg graft (emphasis mine): this command uses mercurial's merge logic copy individual changes other branches without merging branches in history graph. known 'backporting' or 'cherry-picking' . default, graft copy user, date, , description source changesets.
Read more